https://docs.moodle.org/dev/api.php?action=feedcontributions&feedformat=atom&user=Shashitechno 2026-04-21T17:49:43Z User contributions MediaWiki 1.43.5 https://docs.moodle.org/dev/index.php?title=Events_API&diff=44056 2014-03-04T16:44:32Z

<p>Shashitechno: /* Why is a new events system needed? */</p> <hr /> <div>{{Infobox Project<br /> |name = Events 2<br /> |state = Implementation in progress<br /> |tracker = MDL-39797 , MDL-39952, MDL-39846<br /> |discussion = https://moodle.org/mod/forum/discuss.php?d=229425<br /> |assignee = Backend Team<br /> }}<br /> <br /> = What are events? =<br /> <br /> Events are atomic pieces of information describing something that happened in Moodle. Events are primarily the result of user actions, but could also be the result of the [[:en:Cron|cron]] process or administration actions [[:en:Administration via command line|undertaken via the command line]].<br /> <br /> When an action takes place, an event is created by a [[Core APIs|core API]] or [[Plugins|plugin]]. The Events system then disseminates this event information to observers registered for this event. In this way, the events system acts as a communication backbone throughout the Moodle system.<br /> <br /> Event observers can not modify event data or interrupt the dispatching of events, it is a one way communication channel.<br /> <br /> = Why is a new events system needed? =<br /> <br /> The need to improve the Events system was prompted by a need for a richer and more efficient logging system, however the benefits of this improvement will be useful to other parts of Moodle that observe event information.<br /> <br /> * The events need to be more strictly defined if we want to use them for new logging and other advanced use cases. They need to contain a lot more information in a standardised way (such as most fields from current log table and log_actions table).<br /> * Complex data types were allowed in old events which was causing major problems when serialising/storing/unserializing the data.<br /> * The logging and events contain similar information and are triggered at the same places, new events would remove this code duplication. All events should be loggable and all current log entries should be triggered as events.<br /> * The logging system will become an event observer, listening to all events and directing them to logging storage plugins in a controllable way.<br /> * It will be possible to subscribe to '*' event, which would allow a system to potentially observe, and selectively deal with, all events. Current handlers do not get event name which makes this problematic.<br /> * Current event handlers may trigger exceptions during site upgrade which would lead to fatal upgrade problems. The new design eliminates this.<br /> * Failure in handlers blocked dispatching of subsequent events. Instead problems in new observers would be only logged and execution would continue normally.<br /> * Current execution of external handlers during DB transactions blocks other handlers. This would be eliminated by in-memory buffer for external events.<br /> * It would good to have observer priority.<br /> * Nested events are not dispatched sequentially, it would change the order of events received in lower priority handlers.<br /> <br /> = Performance =<br /> Some basic profiling has been conducted.<br /> <br /> There is a general plan to complete pre- and post-implementation testing as development happens. The new system will be imemented in parallel with the old one which should help with comparison of new and old logging and event triggering performance on each page.<br /> <br /> Our aim is to trigger more events and log more information, which is going to impact on performance. We hope to offset that impact by improving log storage, simplifying event dispatching and adding other core performance improvements. The proposed class structure of the base event should allow some new advanced techniques, which may also improve performance in some scenarios.<br /> <br /> More details will be added to this section soon.<br /> <br /> = Events API =<br /> <br /> Each plugin will define the events that it can report (trigger) by extending an abstract base class, once for each possible event. This approach will have several benefits.<br /> ; Events will be active objects<br /> : When they are triggered and possibly after they are reinstantiated (say, when they are retrieved from a log), an event object will be able to provide callback functions for various purposes (such as capability checks).<br /> ; Automatic inclusion<br /> : Event class definitions will be automatically included when needed, without having to maintain lists of known event types. New event definitions can be added without the need to upgrade, only purging of MUC cache is required after adding new observer.<br /> ; Maintainability<br /> : It will be easy to add new events and migrate existing events. Code review will be simplified because there will be less duplication of code when triggering events and all event related information/code will be concentrated in one file in fixed locations.<br /> ; Self documenting<br /> : The behaviour of events will be combined with the definition of events in one place (file). It will be easy for event observer writers to know what events a plugin can trigger. This includes support for autocompletion and code inspection in modern IDEs.<br /> ; Quick, self-validating data structure<br /> : As events are instantiated objects, the PHP processor will validate the structure and type of event classes. This does not ensure data value validity, but does give some assurance of consistency and it also detects common typos.<br /> <br /> == Backwards compatibility and migration ==<br /> <br /> Events:<br /> * Moodle core and standard plugins will replace all calls to the events_trigger() function with new events classes.<br /> * For events that already exist in Moodle 2.5 the additional legacy information should be added to the event data (in properties 'legacyeventname' and 'legacyeventdata'.<br /> * The function events_trigger() will continue working as before, but it will be called automatically after a new event is processed using the 'legacyeventname' and 'legacyeventdata'.<br /> * The legacy events handling code will be maintained separately and will continue being supported in Moodle 2.x. New legacy events will not be added.<br /> * Existing legacy event handlers will be migrated to new event handlers accepting new event class instances.<br /> * More subsystems may be migrated to events-handlers, ex.: gradebook history.<br /> <br /> Logging:<br /> * Function add_to_log() and all logging internals will continue working as before.<br /> * Existing add_to_log() parameters will be migrated inside new events method get_legacy_log_data() and core_event_base::trigger() will call add_to_log() automatically (this will depend on $CFG-&gt;loglifetime setting for performance reasons).<br /> <br /> == Event dispatching and observers ==<br /> <br /> The new event dispatching system is completely separate from the old events code. Original event handlers are now called observers with the description stored in the same db/events.php file, but as a new array with a different format.<br /> <br /> === Event observers ===<br /> <br /> The observers are described in db/events.php in the array $observers, the array is not indexed and contains a list of observers defined as an array with the following properties;<br /> * eventname - event class name or &quot;*&quot; indicating all events. All events must use namespace, ex.: ''\plugintype_pluginname\event\something_happened''.<br /> * callback - PHP callable type.<br /> * includefile - optional. File to be included before calling the observer. Path relative to dirroot.<br /> * priority - optional. Defaults to 0. Observers with higher priority are notified first.<br /> * internal - optional. Defaults to true. Non-internal observers are not called during database transactions, but instead after a successful commit of the transaction.<br /> <br /> &lt;code php&gt;<br /> <br /> $observers = array(<br /> <br /> array(<br /> 'eventname' =&gt; '\core\event\sample_executed',<br /> 'callback' =&gt; 'core_event_sample_observer::observe_one',<br /> ),<br /> <br /> array(<br /> 'eventname' =&gt; '\core\event\sample_executed',<br /> 'callback' =&gt; 'core_event_sample_observer::external_observer',<br /> 'priority' =&gt; 200,<br /> 'internal' =&gt; false,<br /> ),<br /> <br /> array(<br /> 'eventname' =&gt; '*',<br /> 'callback' =&gt; 'core_event_sample_observer::observe_all',<br /> 'includefile' =&gt; null,<br /> 'internal' =&gt; true,<br /> 'priority' =&gt; 9999,<br /> ),<br /> <br /> );<br /> <br /> &lt;/code&gt;<br /> <br /> === Event dispatching ===<br /> <br /> A list of available observers is constructed on the fly directly from all available events.php files. Previously handlers were installed only during installation and upgrade. There is no risk of performance regression because the list is already cached in MUC. Observers get events before installation or any upgrade, however observers are not notified during the initial installation of moodle core tables.<br /> <br /> Developers of observers must make sure that execution does not end with a fatal error under any condition (before install, before upgrade or normal operation). Exceptions are automatically captured, logged in the PHP error log, and notification of other observers continues. Current handlers must not throw any exceptions at any time.<br /> <br /> Observers are notified sequentially in the same order in which events were triggered. This means that events triggered in observers are queued in FIFO buffer and are processed after all observers are notified.<br /> <br /> === Differences from old event handling ===<br /> <br /> # New events contain a lot more structured information.<br /> # New event data must not contain any PHP classes.<br /> # There is separate context cache which may be used when deleting data or for observer performance improvements.<br /> # No database access in new event dispatching code.<br /> # There is no support for cron execution - this eliminates performance problems, simplifies events dispatching and prevents abuse of cron events.<br /> # Events triggered in observers are processed in a different order.<br /> # External events are buffered when a transaction is in progress, instead of being sent to the cron queue.<br /> # It is possible to define multiple observers for one event in one events.php file.<br /> # It is possible to subscribe an observer to all events.<br /> # The new event manager is using frankenstyle autoloading - smaller memory footprint when events are not used on the current page.<br /> <br /> == Triggering events ==<br /> <br /> * All event descriptions are objects extending the \core\event\base class.<br /> * Events are triggered by creating a new instance of the class event and executing $event-&gt;trigger().<br /> * Each event class name is a unique identifier of the event.<br /> * Class names and namespace follow the identifier scheme \'''frankenstyle_component'''\event\'''some_object_action'''. Core events have prefix 'core_'.<br /> * Plugins define each event class in a separate file. File name and location must match the class name, for example: '''plugindir'''/classes/event/'''something_happened'''.php<br /> * The event identifier suffix has the form ''some_object_action'' ('''something_happened''' in the example above) and should follow our standard naming convention. The last word after underscore is automatically parsed as action, the rest of words is object.<br /> <br /> Decision:[[#Verb_list| Recommended verb list]]<br /> <br /> Examples: \core\event\course_completed, \mod_assign\event\submission_commented, \mod_forum\event\post_shared, \mod_forum\event\post_responded, etc.<br /> <br /> * Ideally, it should be possible to trigger an event without gathering additional information for the event. To reduce the cost of data gathering, specifically the cost of database reads, at least the minimal values needed to trigger an event should be already available in variables.<br /> <br /> An example of triggering an event:<br /> &lt;code php&gt;<br /> $event = \mod_myplugin\event\something_happened::create(array('context' =&gt; $context, 'objectid' =&gt; YYY, 'other' =&gt; ZZZ));<br /> // ... code that may add some record snapshots<br /> $event-&gt;trigger();<br /> &lt;/code&gt;<br /> <br /> <br /> === Use of php autoloading ===<br /> <br /> All new OOP APIs in Moodle 2.6 are going to use class auto loading - see [[Automatic class loading]]. New events use PHP strictly defined namespaces which concentrate all events classes in classes/event/ subdirectory.<br /> <br /> === Why separate classes? ===<br /> There were two alternatives proposed on how to define the event structure. The first is a separate class for each event (extending the base class), the other being each event is based on a generic event instance of the base class.<br /> &lt;pre&gt;<br /> Decision: Use separate class for each event.<br /> &lt;/pre&gt;<br /> '''Each plugin creates its own event class for each event'''<br /> <br /> Pros<br /> * Maintainability - It is much easier to review, debug, integrate.<br /> * Self documenting, behaviour is combined with definition.<br /> * It is extremely flexible for plugin developers and core devs too.<br /> * Automatically lists events without being installed - PHPDocs as events documentation.<br /> * It is included only when needed using autoloading.<br /> * Self-validating data structure (by PHP).<br /> * Some developers will find it easier to copy whole class files as templates.<br /> Cons<br /> * Big learning curve for developers without OOP skills (all other new subsystems in Moodle already use OOP, you can not code without these skills any more).<br /> * Some developers may find it harder to copy-and-paste examples because they will need to create new class first and use it afterwards (this can be viewed as a benefit because it forces developers to think more about events).<br /> <br /> <br /> '''Each plugin defines events in a list based on a generic object'''<br /> <br /> Pros<br /> * Easier for some developers (this can be eliminated with developer documentation).<br /> * Some developers think it gives more control of event structure in core (we can easily solve that with private access and validation in the base class).<br /> Cons<br /> * It is not flexible enough. PHP code gives developers more freedom.<br /> * It would not be possible to implement any performance hacks in custom methods. All data would have to be calculated even if it is not used.<br /> * It would be necessary to define access control callbacks in other code.<br /> * It would be harder and slower to integrate legacy logging.<br /> * It would be harder and slower to implement support for legacy events.<br /> * Event observers could not use event class names as reliable identifiers.<br /> * Event object and action could not be parsed from class name, it would have to be stored in event properties every time you trigger event.<br /> * Requires upgrade/install to register an event in DB table with MUC cache. Events could not be triggered earlier.<br /> * The localised descriptions and names would have to be stored as properties, it would not be possible to store them in any definition.<br /> * The implementation of an events infrastructure would be significantly more complex and error prone.<br /> <br /> == Information contained in events ==<br /> <br /> Events have to contain as much information as they can, but this should not affect the performance. That's why part of the information is available in properties, and the rest via methods. This allows for delaying the computation of the data at the time it is really needed, if it ever is.<br /> <br /> === Properties ===<br /> <br /> List of properties that the developer has to pass to the event upon creation, or automatically generated when possible and cost free. Some of those properties not mandatory.<br /> <br /> {| class=&quot;nicetable&quot;<br /> |-<br /> ! Property name<br /> ! Title<br /> ! Type<br /> ! Comment<br /> |-<br /> | ''eventname''<br /> | Event name<br /> | ''static, automatic from class name''<br /> | Automatically computed by copying class name<br /> |-<br /> | ''component''<br /> | Component<br /> | ''static, automatic from top namespace''<br /> | Component declaring the event, automatically computed from class name.<br /> |-<br /> | ''action''<br /> | Action<br /> | ''static, automatic from last word in class name''<br /> | Can be automatically computed from class name.<br /> |-<br /> | ''target''<br /> | target of action<br /> | ''static, automatic from class name''<br /> | Target on which the action is taken, can be automatically computed from class name.<br /> |-<br /> | objecttable<br /> | Database table name<br /> |<br /> | optional database table name where is/was the object stored. Never use a relationship table here.<br /> |-<br /> | objectid<br /> | Object ID<br /> |<br /> | optional id of the object record from objecttable<br /> |-<br /> | '''''crud'''''<br /> | Transaction type<br /> | ''static mandatory''<br /> | One of [crud] letters. Statically declared in the event class method init().<br /> |-<br /> | '''''level''''' / '''''edulevel'''''<br /> | Level<br /> | ''static mandatory''<br /> | Level of educational value of the event. Statically declared in the event class method init(). Changed from '''''level''''' to '''''edulevel''''' in Moodle 2.7 (See below for more details)<br /> |-<br /> | contextid<br /> | Context ID<br /> | mandatory<br /> | <br /> |-<br /> | contextlevel<br /> | Context level<br /> | automatic from context<br /> | This tells you if it was a course, activity, course category, etc.<br /> |-<br /> | contextinstanceid<br /> | Context instanceid<br /> | automatic from context<br /> | Based on context level this may be course id , course module id, course category, etc.<br /> |-<br /> | userid<br /> | User ID<br /> | defaults to current user<br /> | User ID, or 0 when not logged in, or -1 when other (System, CLI, Cron, ...)<br /> |-<br /> | courseid<br /> | Affected course<br /> | defaults to course context from context<br /> | This is used only for contexts at and bellow course level - this can be used to filter events by course (includes all course activities)<br /> |-<br /> | relateduserid<br /> | Affected user<br /> |<br /> | Is this action related to some user? This could be used for some personal timeline view.<br /> |-<br /> | other<br /> | All other data<br /> |<br /> | Any other fields needed for event description - scalars or arrays, must be serialisable using json_encode()<br /> |-<br /> | timecreated<br /> | Time of the event<br /> | automatic<br /> |<br /> |}<br /> <br /> * ''static'': It is the same for all event instances of this class.<br /> * '''mandatory''': Is required in order to trigger the event.<br /> <br /> ==== Level property ====<br /> <br /> The edulevel property helps defining the educational value of the event. It is intentional that the list is limited to only 3 different constants, having too many options would make it harder for developers to pick the right one(s). We also have to keep in mind that this is not supposed to answer all the questions about a particular event, other event properties like the courseid, the context, the component name can be used with the level to get more granular reports.<br /> <br /> Remember that this is event based. If the user that has triggered the event is not really &quot;participating&quot; because he is an admin, or a manager, then it is the job of the report to filter those. The event itself has a static educational level.<br /> <br /> '''Teaching''' (LEVEL_TEACHING: 1)<br /> <br /> Any event/action that is performed by someone (typically a teacher) and has a teaching value (anything that is affecting the learning experience/environment of the students). This should not be combined with &quot;Participating&quot; events.<br /> <br /> Valid events:<br /> <br /> * A teacher grading a student<br /> * A teacher modifying the course settings<br /> * A teacher adding a new section to the course page<br /> * A teacher modifying a module settings<br /> * A teacher adding a page to course<br /> * A teacher leaving a feedback<br /> <br /> INVALID events:<br /> <br /> * A teacher posting in a forum (it might affect the learning experience, but not necessarily, so the teacher is just participating)<br /> <br /> '''Participating''' (LEVEL_PARTICIPATING: 2)<br /> <br /> Any event/action that is performed by a user, that is related (or could be related) to his learning experience.<br /> <br /> Valid events:<br /> <br /> * A user posting to a forum<br /> * A user submitting an assignment<br /> * A user blogging<br /> * A user reading someone's blog<br /> * A user posting a comment<br /> * A user chatting on a chat activity<br /> * A user viewing the course page<br /> * A user deletes a blog post<br /> <br /> INVALID events:<br /> <br /> * A user updating his profile<br /> * A user visiting someone's profile<br /> * A user viewing his /my/ page<br /> * A user sending a message to another one<br /> <br /> '''Other''' (LEVEL_OTHER: 0)<br /> <br /> Any other action, whether they are related to the site administration, or are specific to user. They do not have any educational value.<br /> <br /> === Methods ===<br /> <br /> The computation of this data is not required by default, but can be accessed by any event observer if need be.<br /> <br /> {| class=&quot;nicetable&quot;<br /> |-<br /> ! Method<br /> ! Comment<br /> |-<br /> | get_name()<br /> | Returns localised name of the event, it is the same for all instances.<br /> |-<br /> | get_description()<br /> | Returns localised description of one particular event.<br /> |-<br /> | can_view($user)<br /> | Can the specified user view the event?<br /> |-<br /> | get_url()<br /> | Returns Moodle URL where the event can be observed afterwards.<br /> |-<br /> | get_legacy_eventname()<br /> | Information necessary for event BC.<br /> |-<br /> | get_legacy_eventdata()<br /> | Information necessary for event BC.<br /> |-<br /> | get_legacy_logdata()<br /> | Information necessary for logging BC.<br /> |}<br /> <br /> ===Record caching===<br /> <br /> The standard event data may not contain all the information observers need. The built-in record snapshot support in events allows developers to attach more auxiliary information when triggering events, it may be for example course record, some record that was just deleted, etc. The snapshot is meant to be a full database record, as it will be automatically fetched from get_record_snapshot() if not set previously and assuming the property ''objecttable'' is set. Please be aware that the snapshots are not stored in the event, and cannot be restored.<br /> <br /> &lt;code php&gt;<br /> $event = \core\event\role_assigned::create(<br /> array('context'=&gt;$context, 'objectid'=&gt;$ra-&gt;roleid, 'relateduserid'=&gt;$ra-&gt;userid,<br /> 'other'=&gt;array('id'=&gt;$ra-&gt;id, 'component'=&gt;$ra-&gt;component, 'itemid'=&gt;$ra-&gt;itemid)));<br /> $event-&gt;add_record_snapshot('role_assignments', $ra);<br /> $event-&gt;trigger();<br /> &lt;/code&gt;<br /> <br /> &lt;code php&gt;<br /> $event = \core\event\role_unassigned::create(<br /> array('context'=&gt;$context, 'objectid'=&gt;$ra-&gt;roleid, 'relateduserid'=&gt;$ra-&gt;userid,<br /> 'other'=&gt;array('id'=&gt;$ra-&gt;id, 'component'=&gt;$ra-&gt;component, 'itemid'=&gt;$ra-&gt;itemid)));<br /> $event-&gt;add_record_snapshot('role_assignments', $ra);<br /> $event-&gt;trigger();<br /> &lt;/code&gt;<br /> <br /> The related methods are:<br /> * public function add_record_snapshot($tablename, $record)<br /> * public function get_record_snapshot($tablename, $id)<br /> <br /> === Rejected properties and methods ===<br /> <br /> {| class=&quot;nicetable&quot;<br /> |-<br /> ! Property<br /> ! Title<br /> ! Why<br /> |-<br /> | URL<br /> | relevant page URL<br /> | These URLs can be constructed on the fly from other data, external log plugins may use get_url() method.<br /> |-<br /> | version<br /> | Event specification number<br /> | The event specification should never change<br /> |-<br /> | type<br /> | Type of event (action, error, ...)<br /> | Event are not intended for error logging or debugging.<br /> |-<br /> | actor<br /> | Whether current execution is cron, cli, user, ...<br /> | Impossible to track down at a low level<br /> |-<br /> | severity<br /> | Severity following [http://tools.ietf.org/html/rfc5424#section-6.2.1 logging standards]<br /> | Our logging does not match this, as we will not (at present) log errors<br /> |-<br /> | coursecatname<br /> | Category name<br /> | Might be costly to retrieve for little gain<br /> |-<br /> | coursename<br /> | Course name<br /> | Might be costly to retrieve for little gain<br /> |-<br /> | cmname<br /> | Course module name<br /> | Might be costly to retrieve for little gain<br /> |-<br /> | categoryid <br /> | Course category id<br /> | Categories are a tree structure, we can not identify them by one integer. It would have to be a path.<br /> |-<br /> | cmid <br /> | Course module id<br /> | Can be derived from contextlevel and contextinstanceid<br /> |-<br /> | associatedobject<br /> | Associated object<br /> | Object associated to the main object. Ie: The user to whom a message is sent.<br /> |-<br /> | associatedobjectid<br /> | Associated object ID<br /> | Identifier of the associated object<br /> |-<br /> | realuserid<br /> | Real User ID<br /> | Will be tracked by log plugins only - user who &quot;logged in as&quot;, stores the real user ID<br /> |-<br /> | origin<br /> | Origin of the event<br /> | Will be tracked by log plugins only - CLI, cron, Webservice, ... (optionally with IP address)<br /> |}<br /> <br /> <br /> {| class=&quot;nicetable&quot;<br /> |-<br /> ! Method<br /> ! Comment<br /> ! Why<br /> |-<br /> | get_all_affected_users()<br /> | Returns all the users affected by this event<br /> | It is expensive to fetch all users and it changes in time, so it would be unreliable too. For now we store only one user who is related to each event.<br /> |-<br /> | get_objecturl()<br /> | Returns the URL to view the object<br /> | There is usually only one URL where event changes may be observed. The URL may depend on current user capabilities too.<br /> |-<br /> | get_associatedobjecturl()<br /> | Returns the URL to view the associated object<br /> | No associated user property is present.<br /> |-<br /> | get_currenturl()<br /> | Returns the current URL, uses $PAGE.<br /> | This information may be added by logger, current page info is not part of events data.<br /> |-<br /> | get_useripaddress()<br /> | Returns the User IP address<br /> | Page access method is not part of events API, this should be implemented as logdata properties in loggers.<br /> |}<br /> <br /> == Events naming convention ==<br /> <br /> Clear event names help developers when reading what events are triggered, and defining the events properties when defining the event class.<br /> <br /> Decision: \&lt;component&gt;\event\&lt;some_object&gt;_&lt;verb&gt;<br /> <br /> === Existing events ===<br /> <br /> List of existing events in Moodle code base, along with their 2.5 couterparts.<br /> <br /> {| class=&quot;nicetable&quot;<br /> |-<br /> ! Fully qualified event name<br /> ! 2.5 name<br /> ! Component<br /> ! Object<br /> ! Action (Verb)<br /> ! Comment<br /> |-<br /> | '''assignsubmission_comments\event\comment_created'''<br /> | -<br /> | assignsubmission_comments<br /> | comment<br /> | created<br /> | <br /> |- <br /> | '''assignsubmission_comments\event\comment_deleted'''<br /> | -<br /> | assignsubmission_comments<br /> | comment<br /> | deleted<br /> | <br /> |- <br /> | '''assignsubmission_file\event\assessable_uploaded'''<br /> | assessable_file_uploaded<br /> | assignsubmission_file<br /> | assessable<br /> | uploaded<br /> | To be deprecated MDL-35197<br /> |- <br /> | '''assignsubmission_file\event\submission_created'''<br /> | -<br /> | assignsubmission_file<br /> | submission<br /> | created<br /> | <br /> |- <br /> | '''assignsubmission_file\event\submission_updated'''<br /> | -<br /> | assignsubmission_file<br /> | submission<br /> | updated<br /> | <br /> |- <br /> | '''assignsubmission_onlinetext\event\assessable_uploaded'''<br /> | assessable_content_uploaded<br /> | assignsubmission_onlinetext<br /> | assessable<br /> | uploaded<br /> | <br /> |- <br /> | '''assignsubmission_onlinetext\event\submission_created'''<br /> | -<br /> | assignsubmission_onlinetext<br /> | submission<br /> | created<br /> | <br /> |- <br /> | '''assignsubmission_onlinetext\event\submission_updated'''<br /> | -<br /> | assignsubmission_onlinetext<br /> | submission<br /> | updated<br /> | <br /> |- <br /> | '''block_comments\event\comment_created'''<br /> | -<br /> | block_comments<br /> | comment<br /> | created<br /> | <br /> |- <br /> | '''block_comments\event\comment_deleted'''<br /> | -<br /> | block_comments<br /> | comment<br /> | deleted<br /> | <br /> |- <br /> | '''booktool_exportimscp\event\book_exported'''<br /> | -<br /> | booktool_exportimscp<br /> | book<br /> | exported<br /> | <br /> |- <br /> | '''booktool_print\event\book_printed'''<br /> | -<br /> | booktool_print<br /> | book<br /> | printed<br /> | <br /> |- <br /> | '''booktool_print\event\chapter_printed'''<br /> | -<br /> | booktool_print<br /> | chapter<br /> | printed<br /> | <br /> |- <br /> | '''core\event\assessable_submitted'''<br /> | -<br /> | core<br /> | assessable<br /> | submitted<br /> | Abstract class <br /> |- <br /> | '''core\event\assessable_uploaded'''<br /> | -<br /> | core<br /> | assessable<br /> | uploaded<br /> | Abstract class <br /> |- <br /> | '''core\event\base'''<br /> | -<br /> | core<br /> | base<br /> | ase<br /> | Abstract class <br /> |- <br /> | '''core\event\blog_association_created'''<br /> | -<br /> | core<br /> | blog_association<br /> | created<br /> | <br /> |- <br /> | '''core\event\blog_comment_created'''<br /> | -<br /> | core<br /> | blog_comment<br /> | created<br /> | <br /> |- <br /> | '''core\event\blog_comment_deleted'''<br /> | -<br /> | core<br /> | blog_comment<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\blog_entries_viewed'''<br /> | -<br /> | core<br /> | blog_entries<br /> | viewed<br /> | <br /> |- <br /> | '''core\event\blog_entry_created'''<br /> | blog_entry_added<br /> | core<br /> | blog_entry<br /> | created<br /> | <br /> |- <br /> | '''core\event\blog_entry_deleted'''<br /> | blog_entry_deleted<br /> | core<br /> | blog_entry<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\blog_entry_updated'''<br /> | blog_entry_edited<br /> | core<br /> | blog_entry<br /> | updated<br /> | <br /> |- <br /> | '''core\event\cohort_created'''<br /> | cohort_added<br /> | core<br /> | cohort<br /> | created<br /> | <br /> |- <br /> | '''core\event\cohort_deleted'''<br /> | cohort_deleted<br /> | core<br /> | cohort<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\cohort_member_added'''<br /> | cohort_member_added<br /> | core<br /> | cohort_member<br /> | added<br /> | <br /> |- <br /> | '''core\event\cohort_member_removed'''<br /> | cohort_member_removed<br /> | core<br /> | cohort_member<br /> | removed<br /> | <br /> |- <br /> | '''core\event\cohort_updated'''<br /> | cohort_updated<br /> | core<br /> | cohort<br /> | updated<br /> | <br /> |- <br /> | '''core\event\comment_created'''<br /> | -<br /> | core<br /> | comment<br /> | created<br /> | Abstract class <br /> |- <br /> | '''core\event\comment_deleted'''<br /> | -<br /> | core<br /> | comment<br /> | deleted<br /> | Abstract class <br /> |- <br /> | '''core\event\comments_viewed'''<br /> | -<br /> | core<br /> | comments<br /> | viewed<br /> | Abstract class <br /> |- <br /> | '''core\event\content_viewed'''<br /> | -<br /> | core<br /> | content<br /> | viewed<br /> | Abstract class <br /> |- <br /> | '''core\event\course_category_created'''<br /> | -<br /> | core<br /> | course_category<br /> | created<br /> | <br /> |- <br /> | '''core\event\course_category_deleted'''<br /> | course_category_deleted<br /> | core<br /> | course_category<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\course_category_updated'''<br /> | -<br /> | core<br /> | course_category<br /> | updated<br /> | <br /> |- <br /> | '''core\event\course_completed'''<br /> | course_completed<br /> | core<br /> | course<br /> | completed<br /> | <br /> |- <br /> | '''core\event\course_completion_updated'''<br /> | -<br /> | core<br /> | course_completion<br /> | updated<br /> | <br /> |- <br /> | '''core\event\course_content_deleted'''<br /> | course_content_removed<br /> | core<br /> | course_content<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\course_created'''<br /> | course_created<br /> | core<br /> | course<br /> | created<br /> | <br /> |- <br /> | '''core\event\course_deleted'''<br /> | course_deleted<br /> | core<br /> | course<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\course_module_completion_updated'''<br /> | activity_completion_changed<br /> | core<br /> | course_module_completion<br /> | updated<br /> | <br /> |- <br /> | '''core\event\course_module_created'''<br /> | mod_created<br /> | core<br /> | course_module<br /> | created<br /> | <br /> |- <br /> | '''core\event\course_module_deleted'''<br /> | mod_deleted<br /> | core<br /> | course_module<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\course_module_instance_list_viewed'''<br /> | -<br /> | core<br /> | course_module_instance_list<br /> | viewed<br /> | Abstract class <br /> |- <br /> | '''core\event\course_module_instances_list_viewed'''<br /> | -<br /> | core<br /> | course_module_instances_list<br /> | viewed<br /> | Abstract class <br /> |- <br /> | '''core\event\course_module_updated'''<br /> | mod_updated<br /> | core<br /> | course_module<br /> | updated<br /> | <br /> |- <br /> | '''core\event\course_module_viewed'''<br /> | -<br /> | core<br /> | course_module<br /> | viewed<br /> | Abstract class <br /> |- <br /> | '''core\event\course_reset_ended'''<br /> | -<br /> | core<br /> | course_reset<br /> | ended<br /> | <br /> |- <br /> | '''core\event\course_reset_started'''<br /> | -<br /> | core<br /> | course_reset<br /> | started<br /> | <br /> |- <br /> | '''core\event\course_restored'''<br /> | course_restored<br /> | core<br /> | course<br /> | restored<br /> | <br /> |- <br /> | '''core\event\course_section_updated'''<br /> | -<br /> | core<br /> | course_section<br /> | updated<br /> | <br /> |- <br /> | '''core\event\course_updated'''<br /> | course_updated<br /> | core<br /> | course<br /> | updated<br /> | <br /> |- <br /> | '''core\event\email_failed'''<br /> | -<br /> | core<br /> | email<br /> | failed<br /> | <br /> |- <br /> | '''core\event\group_created'''<br /> | groups_group_created<br /> | core<br /> | group<br /> | created<br /> | <br /> |- <br /> | '''core\event\group_deleted'''<br /> | groups_group_deleted<br /> | core<br /> | group<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\group_member_added'''<br /> | groups_member_added<br /> | core<br /> | group_member<br /> | added<br /> | <br /> |- <br /> | '''core\event\group_member_removed'''<br /> | groups_member_removed<br /> | core<br /> | group_member<br /> | removed<br /> | <br /> |- <br /> | '''core\event\group_updated'''<br /> | groups_group_updated<br /> | core<br /> | group<br /> | updated<br /> | <br /> |- <br /> | '''core\event\grouping_created'''<br /> | groups_grouping_created<br /> | core<br /> | grouping<br /> | created<br /> | <br /> |- <br /> | '''core\event\grouping_deleted'''<br /> | groups_grouping_deleted<br /> | core<br /> | grouping<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\grouping_updated'''<br /> | groups_grouping_updated<br /> | core<br /> | grouping<br /> | updated<br /> | <br /> |- <br /> | '''core\event\manager'''<br /> | -<br /> | core<br /> | manager<br /> | anager<br /> | <br /> |- <br /> | '''core\event\mnet_access_control_created'''<br /> | -<br /> | core<br /> | mnet_access_control<br /> | created<br /> | <br /> |- <br /> | '''core\event\mnet_access_control_updated'''<br /> | -<br /> | core<br /> | mnet_access_control<br /> | updated<br /> | <br /> |- <br /> | '''core\event\note_created'''<br /> | -<br /> | core<br /> | note<br /> | created<br /> | <br /> |- <br /> | '''core\event\note_deleted'''<br /> | -<br /> | core<br /> | note<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\note_updated'''<br /> | -<br /> | core<br /> | note<br /> | updated<br /> | <br /> |- <br /> | '''core\event\notes_viewed'''<br /> | -<br /> | core<br /> | notes<br /> | viewed<br /> | <br /> |- <br /> | '''core\event\role_allow_assign_updated'''<br /> | -<br /> | core<br /> | role_allow_assign<br /> | updated<br /> | <br /> |- <br /> | '''core\event\role_allow_override_updated'''<br /> | -<br /> | core<br /> | role_allow_override<br /> | updated<br /> | <br /> |- <br /> | '''core\event\role_allow_switch_updated'''<br /> | -<br /> | core<br /> | role_allow_switch<br /> | updated<br /> | <br /> |- <br /> | '''core\event\role_assigned'''<br /> | role_assigned<br /> | core<br /> | role<br /> | assigned<br /> | <br /> |- <br /> | '''core\event\role_capabilities_updated'''<br /> | -<br /> | core<br /> | role_capabilities<br /> | updated<br /> | <br /> |- <br /> | '''core\event\role_deleted'''<br /> | -<br /> | core<br /> | role<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\role_unassigned'''<br /> | role_unassigned<br /> | core<br /> | role<br /> | unassigned<br /> | <br /> |- <br /> | '''core\event\user_created'''<br /> | user_created<br /> | core<br /> | user<br /> | created<br /> | <br /> |- <br /> | '''core\event\user_deleted'''<br /> | user_deleted<br /> | core<br /> | user<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\user_enrolment_created'''<br /> | user_enrolled<br /> | core<br /> | user_enrolment<br /> | created<br /> | <br /> |- <br /> | '''core\event\user_enrolment_deleted'''<br /> | user_unenrolled<br /> | core<br /> | user_enrolment<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\user_enrolment_updated'''<br /> | user_enrol_modified<br /> | core<br /> | user_enrolment<br /> | updated<br /> | <br /> |- <br /> | '''core\event\user_list_viewed'''<br /> | -<br /> | core<br /> | user_list<br /> | viewed<br /> | <br /> |- <br /> | '''core\event\user_loggedin'''<br /> | -<br /> | core<br /> | user<br /> | loggedin<br /> | <br /> |- <br /> | '''core\event\user_loggedinas'''<br /> | -<br /> | core<br /> | user<br /> | loggedinas<br /> | <br /> |- <br /> | '''core\event\user_loggedout'''<br /> | user_logout<br /> | core<br /> | user<br /> | loggedout<br /> | <br /> |- <br /> | '''core\event\user_login_failed'''<br /> | -<br /> | core<br /> | user_login<br /> | failed<br /> | <br /> |- <br /> | '''core\event\user_profile_viewed'''<br /> | -<br /> | core<br /> | user_profile<br /> | viewed<br /> | <br /> |- <br /> | '''core\event\user_updated'''<br /> | user_updated<br /> | core<br /> | user<br /> | updated<br /> | <br /> |- <br /> | '''core\event\webservice_function_called'''<br /> | -<br /> | core<br /> | webservice_function<br /> | called<br /> | <br /> |- <br /> | '''core\event\webservice_login_failed'''<br /> | -<br /> | core<br /> | webservice_login<br /> | failed<br /> | <br /> |- <br /> | '''core\event\webservice_service_created'''<br /> | -<br /> | core<br /> | webservice_service<br /> | created<br /> | <br /> |- <br /> | '''core\event\webservice_service_deleted'''<br /> | -<br /> | core<br /> | webservice_service<br /> | deleted<br /> | <br /> |- <br /> | '''core\event\webservice_service_updated'''<br /> | -<br /> | core<br /> | webservice_service<br /> | updated<br /> | <br /> |- <br /> | '''core\event\webservice_service_user_added'''<br /> | -<br /> | core<br /> | webservice_service_user<br /> | added<br /> | <br /> |- <br /> | '''core\event\webservice_service_user_removed'''<br /> | -<br /> | core<br /> | webservice_service_user<br /> | removed<br /> | <br /> |- <br /> | '''core\event\webservice_token_created'''<br /> | -<br /> | core<br /> | webservice_token<br /> | created<br /> | <br /> |- <br /> | '''core\event\webservice_token_sent'''<br /> | -<br /> | core<br /> | webservice_token<br /> | sent<br /> | <br /> |- <br /> | '''logstore_legacy\event\legacy_logged'''<br /> | -<br /> | logstore_legacy<br /> | legacy<br /> | logged<br /> | <br /> |- <br /> | '''mod_assign\event\all_submissions_downloaded'''<br /> | -<br /> | mod_assign<br /> | all_submissions<br /> | downloaded<br /> | <br /> |- <br /> | '''mod_assign\event\assessable_submitted'''<br /> | assessable_submitted<br /> | mod_assign<br /> | assessable<br /> | submitted<br /> | <br /> |- <br /> | '''mod_assign\event\extension_granted'''<br /> | -<br /> | mod_assign<br /> | extension<br /> | granted<br /> | <br /> |- <br /> | '''mod_assign\event\identities_revealed'''<br /> | -<br /> | mod_assign<br /> | identities<br /> | revealed<br /> | <br /> |- <br /> | '''mod_assign\event\marker_updated'''<br /> | -<br /> | mod_assign<br /> | marker<br /> | updated<br /> | <br /> |- <br /> | '''mod_assign\event\statement_accepted'''<br /> | -<br /> | mod_assign<br /> | statement<br /> | accepted<br /> | <br /> |- <br /> | '''mod_assign\event\submission_created'''<br /> | -<br /> | mod_assign<br /> | submission<br /> | created<br /> | Abstract class <br /> |- <br /> | '''mod_assign\event\submission_duplicated'''<br /> | -<br /> | mod_assign<br /> | submission<br /> | duplicated<br /> | <br /> |- <br /> | '''mod_assign\event\submission_graded'''<br /> | -<br /> | mod_assign<br /> | submission<br /> | graded<br /> | <br /> |- <br /> | '''mod_assign\event\submission_locked'''<br /> | -<br /> | mod_assign<br /> | submission<br /> | locked<br /> | <br /> |- <br /> | '''mod_assign\event\submission_status_updated'''<br /> | -<br /> | mod_assign<br /> | submission_status<br /> | updated<br /> | <br /> |- <br /> | '''mod_assign\event\submission_unlocked'''<br /> | -<br /> | mod_assign<br /> | submission<br /> | unlocked<br /> | <br /> |- <br /> | '''mod_assign\event\submission_updated'''<br /> | -<br /> | mod_assign<br /> | submission<br /> | updated<br /> | Abstract class <br /> |- <br /> | '''mod_assign\event\workflow_state_updated'''<br /> | -<br /> | mod_assign<br /> | workflow_state<br /> | updated<br /> | <br /> |- <br /> | '''mod_book\event\chapter_created'''<br /> | -<br /> | mod_book<br /> | chapter<br /> | created<br /> | <br /> |- <br /> | '''mod_book\event\chapter_deleted'''<br /> | -<br /> | mod_book<br /> | chapter<br /> | deleted<br /> | <br /> |- <br /> | '''mod_book\event\chapter_updated'''<br /> | -<br /> | mod_book<br /> | chapter<br /> | updated<br /> | <br /> |- <br /> | '''mod_book\event\chapter_viewed'''<br /> | -<br /> | mod_book<br /> | chapter<br /> | viewed<br /> | <br /> |- <br /> | '''mod_book\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_book<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_book\event\course_module_viewed'''<br /> | -<br /> | mod_book<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_chat\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_chat<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_chat\event\message_sent'''<br /> | -<br /> | mod_chat<br /> | message<br /> | sent<br /> | <br /> |- <br /> | '''mod_chat\event\sessions_viewed'''<br /> | -<br /> | mod_chat<br /> | sessions<br /> | viewed<br /> | <br /> |- <br /> | '''mod_choice\event\answer_submitted'''<br /> | -<br /> | mod_choice<br /> | answer<br /> | submitted<br /> | <br /> |- <br /> | '''mod_choice\event\answer_updated'''<br /> | -<br /> | mod_choice<br /> | answer<br /> | updated<br /> | <br /> |- <br /> | '''mod_choice\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_choice<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_choice\event\course_module_viewed'''<br /> | -<br /> | mod_choice<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_choice\event\report_viewed'''<br /> | -<br /> | mod_choice<br /> | report<br /> | viewed<br /> | <br /> |- <br /> | '''mod_data\event\comment_created'''<br /> | -<br /> | mod_data<br /> | comment<br /> | created<br /> | <br /> |- <br /> | '''mod_data\event\comment_deleted'''<br /> | -<br /> | mod_data<br /> | comment<br /> | deleted<br /> | <br /> |- <br /> | '''mod_data\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_data<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_data\event\course_module_viewed'''<br /> | -<br /> | mod_data<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_data\event\field_created'''<br /> | -<br /> | mod_data<br /> | field<br /> | created<br /> | <br /> |- <br /> | '''mod_data\event\field_deleted'''<br /> | -<br /> | mod_data<br /> | field<br /> | deleted<br /> | <br /> |- <br /> | '''mod_data\event\field_updated'''<br /> | -<br /> | mod_data<br /> | field<br /> | updated<br /> | <br /> |- <br /> | '''mod_data\event\record_created'''<br /> | -<br /> | mod_data<br /> | record<br /> | created<br /> | <br /> |- <br /> | '''mod_data\event\record_deleted'''<br /> | -<br /> | mod_data<br /> | record<br /> | deleted<br /> | <br /> |- <br /> | '''mod_data\event\record_updated'''<br /> | -<br /> | mod_data<br /> | record<br /> | updated<br /> | <br /> |- <br /> | '''mod_data\event\template_updated'''<br /> | -<br /> | mod_data<br /> | template<br /> | updated<br /> | <br /> |- <br /> | '''mod_data\event\template_viewed'''<br /> | -<br /> | mod_data<br /> | template<br /> | viewed<br /> | <br /> |- <br /> | '''mod_feedback\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_feedback<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_feedback\event\course_module_viewed'''<br /> | -<br /> | mod_feedback<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_feedback\event\response_deleted'''<br /> | -<br /> | mod_feedback<br /> | response<br /> | deleted<br /> | <br /> |- <br /> | '''mod_feedback\event\response_submitted'''<br /> | -<br /> | mod_feedback<br /> | response<br /> | submitted<br /> | <br /> |- <br /> | '''mod_folder\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_folder<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_folder\event\course_module_viewed'''<br /> | -<br /> | mod_folder<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_folder\event\folder_updated'''<br /> | -<br /> | mod_folder<br /> | folder<br /> | updated<br /> | <br /> |- <br /> | '''mod_forum\event\assessable_uploaded'''<br /> | assessable_content_uploaded<br /> | mod_forum<br /> | assessable<br /> | uploaded<br /> | <br /> |- <br /> | '''mod_forum\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_forum<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_forum\event\course_searched'''<br /> | -<br /> | mod_forum<br /> | course<br /> | searched<br /> | <br /> |- <br /> | '''mod_forum\event\discussion_created'''<br /> | -<br /> | mod_forum<br /> | discussion<br /> | created<br /> | <br /> |- <br /> | '''mod_forum\event\discussion_deleted'''<br /> | -<br /> | mod_forum<br /> | discussion<br /> | deleted<br /> | <br /> |- <br /> | '''mod_forum\event\discussion_moved'''<br /> | -<br /> | mod_forum<br /> | discussion<br /> | moved<br /> | <br /> |- <br /> | '''mod_forum\event\discussion_updated'''<br /> | -<br /> | mod_forum<br /> | discussion<br /> | updated<br /> | <br /> |- <br /> | '''mod_forum\event\discussion_viewed'''<br /> | -<br /> | mod_forum<br /> | discussion<br /> | viewed<br /> | <br /> |- <br /> | '''mod_forum\event\forum_viewed'''<br /> | -<br /> | mod_forum<br /> | forum<br /> | viewed<br /> | <br /> |- <br /> | '''mod_forum\event\post_created'''<br /> | -<br /> | mod_forum<br /> | post<br /> | created<br /> | <br /> |- <br /> | '''mod_forum\event\post_deleted'''<br /> | -<br /> | mod_forum<br /> | post<br /> | deleted<br /> | <br /> |- <br /> | '''mod_forum\event\post_updated'''<br /> | -<br /> | mod_forum<br /> | post<br /> | updated<br /> | <br /> |- <br /> | '''mod_forum\event\readtracking_disabled'''<br /> | -<br /> | mod_forum<br /> | readtracking<br /> | disabled<br /> | <br /> |- <br /> | '''mod_forum\event\readtracking_enabled'''<br /> | -<br /> | mod_forum<br /> | readtracking<br /> | enabled<br /> | <br /> |- <br /> | '''mod_forum\event\subscribers_viewed'''<br /> | -<br /> | mod_forum<br /> | subscribers<br /> | viewed<br /> | <br /> |- <br /> | '''mod_forum\event\subscription_created'''<br /> | -<br /> | mod_forum<br /> | subscription<br /> | created<br /> | <br /> |- <br /> | '''mod_forum\event\subscription_deleted'''<br /> | -<br /> | mod_forum<br /> | subscription<br /> | deleted<br /> | <br /> |- <br /> | '''mod_forum\event\userreport_viewed'''<br /> | -<br /> | mod_forum<br /> | userreport<br /> | viewed<br /> | <br /> |- <br /> | '''mod_glossary\event\comment_created'''<br /> | -<br /> | mod_glossary<br /> | comment<br /> | created<br /> | <br /> |- <br /> | '''mod_glossary\event\comment_deleted'''<br /> | -<br /> | mod_glossary<br /> | comment<br /> | deleted<br /> | <br /> |- <br /> | '''mod_lesson\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_lesson<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_lesson\event\course_module_viewed'''<br /> | -<br /> | mod_lesson<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_lesson\event\essay_assessed'''<br /> | -<br /> | mod_lesson<br /> | essay<br /> | assessed<br /> | <br /> |- <br /> | '''mod_lesson\event\essay_attempt_viewed'''<br /> | -<br /> | mod_lesson<br /> | essay_attempt<br /> | viewed<br /> | <br /> |- <br /> | '''mod_lesson\event\highscore_added'''<br /> | -<br /> | mod_lesson<br /> | highscore<br /> | added<br /> | <br /> |- <br /> | '''mod_lesson\event\highscores_viewed'''<br /> | -<br /> | mod_lesson<br /> | highscores<br /> | viewed<br /> | <br /> |- <br /> | '''mod_lesson\event\lesson_ended'''<br /> | -<br /> | mod_lesson<br /> | lesson<br /> | ended<br /> | <br /> |- <br /> | '''mod_lesson\event\lesson_started'''<br /> | -<br /> | mod_lesson<br /> | lesson<br /> | started<br /> | <br /> |- <br /> | '''mod_lti\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_lti<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_lti\event\course_module_viewed'''<br /> | -<br /> | mod_lti<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_lti\event\unknown_service_api_called'''<br /> | lti_unknown_service_api_call<br /> | mod_lti<br /> | unknown_service_api<br /> | called<br /> | <br /> |- <br /> | '''mod_page\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_page<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_page\event\course_module_viewed'''<br /> | -<br /> | mod_page<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_quiz\event\attempt_abandoned'''<br /> | quiz_attempt_abandoned<br /> | mod_quiz<br /> | attempt<br /> | abandoned<br /> | <br /> |- <br /> | '''mod_quiz\event\attempt_becameoverdue'''<br /> | quiz_attempt_overdue<br /> | mod_quiz<br /> | attempt<br /> | becameoverdue<br /> | <br /> |- <br /> | '''mod_quiz\event\attempt_started'''<br /> | quiz_attempt_started<br /> | mod_quiz<br /> | attempt<br /> | started<br /> | <br /> |- <br /> | '''mod_quiz\event\attempt_submitted'''<br /> | quiz_attempt_submitted<br /> | mod_quiz<br /> | attempt<br /> | submitted<br /> | <br /> |- <br /> | '''mod_resource\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_resource<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_resource\event\course_module_viewed'''<br /> | -<br /> | mod_resource<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_scorm\event\attempt_deleted'''<br /> | -<br /> | mod_scorm<br /> | attempt<br /> | deleted<br /> | <br /> |- <br /> | '''mod_scorm\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_scorm<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_scorm\event\course_module_viewed'''<br /> | -<br /> | mod_scorm<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_scorm\event\interactions_viewed'''<br /> | -<br /> | mod_scorm<br /> | interactions<br /> | viewed<br /> | <br /> |- <br /> | '''mod_scorm\event\report_viewed'''<br /> | -<br /> | mod_scorm<br /> | report<br /> | viewed<br /> | <br /> |- <br /> | '''mod_scorm\event\sco_launched'''<br /> | -<br /> | mod_scorm<br /> | sco<br /> | launched<br /> | <br /> |- <br /> | '''mod_scorm\event\tracks_viewed'''<br /> | -<br /> | mod_scorm<br /> | tracks<br /> | viewed<br /> | <br /> |- <br /> | '''mod_scorm\event\user_report_viewed'''<br /> | -<br /> | mod_scorm<br /> | user_report<br /> | viewed<br /> | <br /> |- <br /> | '''mod_url\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_url<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_url\event\course_module_viewed'''<br /> | -<br /> | mod_url<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_wiki\event\comment_created'''<br /> | -<br /> | mod_wiki<br /> | comment<br /> | created<br /> | <br /> |- <br /> | '''mod_wiki\event\comment_deleted'''<br /> | -<br /> | mod_wiki<br /> | comment<br /> | deleted<br /> | <br /> |- <br /> | '''mod_wiki\event\comments_viewed'''<br /> | -<br /> | mod_wiki<br /> | comments<br /> | viewed<br /> | <br /> |- <br /> | '''mod_wiki\event\course_module_instance_list_viewed'''<br /> | -<br /> | mod_wiki<br /> | course_module_instance_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_wiki\event\course_module_viewed'''<br /> | -<br /> | mod_wiki<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_wiki\event\page_created'''<br /> | -<br /> | mod_wiki<br /> | page<br /> | created<br /> | <br /> |- <br /> | '''mod_wiki\event\page_deleted'''<br /> | -<br /> | mod_wiki<br /> | page<br /> | deleted<br /> | <br /> |- <br /> | '''mod_wiki\event\page_diff_viewed'''<br /> | -<br /> | mod_wiki<br /> | page_diff<br /> | viewed<br /> | <br /> |- <br /> | '''mod_wiki\event\page_history_viewed'''<br /> | -<br /> | mod_wiki<br /> | page_history<br /> | viewed<br /> | <br /> |- <br /> | '''mod_wiki\event\page_locks_deleted'''<br /> | -<br /> | mod_wiki<br /> | page_locks<br /> | deleted<br /> | <br /> |- <br /> | '''mod_wiki\event\page_map_viewed'''<br /> | -<br /> | mod_wiki<br /> | page_map<br /> | viewed<br /> | <br /> |- <br /> | '''mod_wiki\event\page_updated'''<br /> | -<br /> | mod_wiki<br /> | page<br /> | updated<br /> | <br /> |- <br /> | '''mod_wiki\event\page_version_deleted'''<br /> | -<br /> | mod_wiki<br /> | page_version<br /> | deleted<br /> | <br /> |- <br /> | '''mod_wiki\event\page_version_restored'''<br /> | -<br /> | mod_wiki<br /> | page_version<br /> | restored<br /> | <br /> |- <br /> | '''mod_wiki\event\page_version_viewed'''<br /> | -<br /> | mod_wiki<br /> | page_version<br /> | viewed<br /> | <br /> |- <br /> | '''mod_wiki\event\page_viewed'''<br /> | -<br /> | mod_wiki<br /> | page<br /> | viewed<br /> | <br /> |- <br /> | '''mod_workshop\event\assessable_uploaded'''<br /> | assessable_content_uploaded<br /> | mod_workshop<br /> | assessable<br /> | uploaded<br /> | <br /> |- <br /> | '''mod_workshop\event\assessment_evaluated'''<br /> | -<br /> | mod_workshop<br /> | assessment<br /> | evaluated<br /> | <br /> |- <br /> | '''mod_workshop\event\assessment_evaluations_reset'''<br /> | -<br /> | mod_workshop<br /> | assessment_evaluations<br /> | reset<br /> | <br /> |- <br /> | '''mod_workshop\event\assessment_reevaluated'''<br /> | -<br /> | mod_workshop<br /> | assessment<br /> | reevaluated<br /> | <br /> |- <br /> | '''mod_workshop\event\course_module_viewed'''<br /> | workshop_viewed<br /> | mod_workshop<br /> | course_module<br /> | viewed<br /> | <br /> |- <br /> | '''mod_workshop\event\instances_list_viewed'''<br /> | -<br /> | mod_workshop<br /> | instances_list<br /> | viewed<br /> | <br /> |- <br /> | '''mod_workshop\event\phase_switched'''<br /> | -<br /> | mod_workshop<br /> | phase<br /> | switched<br /> | <br /> |- <br /> | '''mod_workshop\event\submission_assessed'''<br /> | -<br /> | mod_workshop<br /> | submission<br /> | assessed<br /> | <br /> |- <br /> | '''mod_workshop\event\submission_created'''<br /> | -<br /> | mod_workshop<br /> | submission<br /> | created<br /> | <br /> |- <br /> | '''mod_workshop\event\submission_reassessed'''<br /> | -<br /> | mod_workshop<br /> | submission<br /> | reassessed<br /> | <br /> |- <br /> | '''mod_workshop\event\submission_updated'''<br /> | -<br /> | mod_workshop<br /> | submission<br /> | updated<br /> | <br /> |- <br /> | '''mod_workshop\event\submission_viewed'''<br /> | -<br /> | mod_workshop<br /> | submission<br /> | viewed<br /> | <br /> |- <br /> | '''report_log\event\content_viewed'''<br /> | -<br /> | report_log<br /> | content<br /> | viewed<br /> | <br /> |- <br /> | '''report_loglive\event\content_viewed'''<br /> | -<br /> | report_loglive<br /> | content<br /> | viewed<br /> | <br /> |- <br /> | '''report_outline\event\content_viewed'''<br /> | -<br /> | report_outline<br /> | content<br /> | viewed<br /> | <br /> |- <br /> | '''report_participation\event\content_viewed'''<br /> | -<br /> | report_participation<br /> | content<br /> | viewed<br /> | <br /> |- <br /> | '''report_stats\event\content_viewed'''<br /> | -<br /> | report_stats<br /> | content<br /> | viewed<br /> | <br /> |- <br /> |}<br /> <br /> === Verb list ===<br /> All events must use a verb from this list. New verbs should be added to this list if required.<br /> {| class=&quot;nicetable&quot;<br /> |-<br /> !verb<br /> !Explanation<br /> !Source<br /> |-<br /> |abandoned<br /> |When a attempt is abandoned by user (Quiz attempt)<br /> |Moodle<br /> |-<br /> |accepted<br /> |Example: Accepting a statement when submitting an assignment.<br /> |Moodle<br /> |-<br /> |added<br /> | Used to represent &quot;something that already exists is now part of/bound to another entity&quot;. Examples: &quot;Admin added role to user X&quot;, &quot;Admin added user X to group A&quot;. Wrong example: &quot;User added course in category&quot; because it is a 'move' action, except if a course can be part of multiple categories. The good examples work because: A user can have multiple roles, a user can be in multiple groups.<br /> |Moodle<br /> |-<br /> |answered<br /> | Indicates the actor responded to a Question<br /> |Tincan<br /> |-<br /> |assessed<br /> | Some submitted material has been assessed<br /> |Moodle<br /> |-<br /> |assigned<br /> | Assign some privilege or role to user.<br /> |Moodle<br /> |-<br /> |attempted<br /> | Trying to do an activity. Example: attempting a Math class.<br /> |Tincan<br /> |-<br /> |awarded<br /> | ex:-teacher awarded student a badge.<br /> |Moodle<br /> |-<br /> |backedup<br /> | When a backup has been performed.<br /> |Moodle <br /> |-<br /> |becomeoverdue<br /> | When an activity is overdue Example: Quiz attempt is overdue<br /> |Moodle <br /> |-<br /> |called<br /> | When a call to something is made like an API @see unknown_service_api_called.php<br /> |Moodle<br /> |-<br /> |commented<br /> |Offered an opinion or written experience of the activity. Can be used with the learner as the actor or a system as an actor. Comments can be sent from either party with the idea that the other will read and react to the content.<br /> |Tincan<br /> |-<br /> |completed<br /> |To experience the activity in its entirety. Used to affirm the completion of content. This can be simply experiencing all the content, be tied to objectives or interactions, or determined in any other way. Any content that has been initialized, but not yet completed, should be considered incomplete. There is no verb to 'incomplete' an activity, one would void the statement which completes the activity.&quot;<br /> |Tincan<br /> |-<br /> |created<br /> |Used to represent &quot;something new has been created&quot;.<br /> |Moodle<br /> |-<br /> |deleted<br /> |Used to indicate the object in context was deleted.<br /> |Moodle<br /> |-<br /> |disabled<br /> |When an activity is disabled. Example: forum read tracking disabled.<br /> |Moodle<br /> |-<br /> |downloaded<br /> |When a user download file from user. Example submission/report downloaded.<br /> |Moodle<br /> |-<br /> |duplicated<br /> |For something that has been copied.<br /> |Moodle<br /> |-<br /> |enabled<br /> |When some setting is enabled. Example: forum read tracking enabled.<br /> |Moodle<br /> |-<br /> |ended<br /> |When a process ends. Example: Lesson ended or course reset ended.<br /> |Moodle<br /> |-<br /> |evaluated<br /> |Material has been evaluated.<br /> |Moodle<br /> |-<br /> |exported<br /> |When a report is exported in certain format.<br /> |Moodle<br /> |-<br /> |failed<br /> |Learner did not perform the activity to a level of pre-determined satisfaction. Used to affirm the lack of success a learner experienced within the learning content in relation to a threshold. If the user performed below the minimum to the level of this threshold, the content is 'failed'. The opposite of 'passed'. This is also used in case when message sending is failed.<br /> |Tincan<br /> |-<br /> |graded<br /> |Used to represent an activity was graded.<br /> |Moodle<br /> |-<br /> |granted<br /> |User is granted some extension or capability. Example: extension granted for submission.<br /> |Moodle<br /> |-<br /> |imported<br /> |The act of moving an object into another location or system.<br /> |Tincan<br /> |-<br /> |launched<br /> |When an external object is launched. Try consider started if there is related stopped event.<br /> |Moodle<br /> |-<br /> |locked<br /> |When an activity is locked. Should have a related unlocked event.<br /> |Moodle<br /> |-<br /> |loggedin/loggedout<br /> | For login and logout.<br /> |Moodle<br /> |-<br /> |loggedinas<br /> | If user is logged in as different user. This is used by only one event (user_loggedinas). Adding this verb makes event name more clear, then using loggedin verb.<br /> |Moodle<br /> |-<br /> |locked<br /> |<br /> |Moodle<br /> |-<br /> |moved<br /> | Used to indicate the object in context was moved.<br /> |Moodle<br /> |-<br /> |passed<br /> |Used to affirm the success a learner experienced within the learning content in relation to a threshold. If the user performed at a minimum to the level of this threshold, the content is 'passed'. The opposite of 'failed'.<br /> |Tincan<br /> |-<br /> |printed<br /> |Something is printed.<br /> |Moodle<br /> |-<br /> |reassessed<br /> |Submitted material has been assessed again.<br /> |Moodle<br /> |-<br /> |reevaluated<br /> |Material has been evaluated again.<br /> |Moodle<br /> |-<br /> |removed<br /> |By opposition to &quot;Added&quot;. This does not mean that the object has been deleted, but removed from the entity, or not bound to it any more.<br /> |Moodle<br /> |-<br /> |reset<br /> |Sets one or more properties back to the default value.<br /> |Moodle<br /> |-<br /> |restored<br /> |When restoring a backup. Rolling back to a previous state.<br /> |Moodle<br /> |-<br /> |revealed<br /> |Some identity is revealed. Example: Identities revealed after blind marking.<br /> |Moodle<br /> |-<br /> |searched<br /> |Something is searched. Example: searched in course.<br /> |Moodle<br /> |-<br /> |sent<br /> |Message sent.<br /> |Moodle<br /> |-<br /> |started<br /> |Some activity started<br /> |Moodle<br /> |-<br /> |submitted<br /> |This is very close to &quot;Attempted&quot;. Depends on context which one should be used. For example:- &quot;Admin submitted a form. Student attempted a quiz.&quot; is correct, however some cases might not be as clear as the previous example. We can say both &quot;Student submitted an assignment&quot; or &quot;student attempted an assignment&quot;. We need to make the difference clear.<br /> |Moodle<br /> |-<br /> |suspended<br /> | Suspend something. (example a user)<br /> | Tincan (However the context is different)<br /> |-<br /> |switched<br /> |Something has been switched. For example:- The workshop phase has been switched to assessment&quot;<br /> |Moodle<br /> |-<br /> |unassigned<br /> |As opposed to assigned. When some role is unassigned.<br /> |Moodle<br /> |-<br /> |unlocked<br /> |<br /> |Moodle<br /> |-<br /> |upgraded<br /> |Something was upgraded, some module probably<br /> |Moodle<br /> |-<br /> |updated<br /> |Used to indicate the object in context was updated. Simple example is &quot;Admin updated course xyz&quot;.<br /> |Moodle<br /> |-<br /> |uploaded<br /> |When an assignment is uploaded.<br /> |Moodle<br /> |-<br /> |viewed<br /> |Something has been viewed. For example:- &quot;Student viewed chapter 1 of book 1.&quot;<br /> |Moodle<br /> |-<br /> |}<br /> <br /> === Rules ===<br /> <br /> ==== Use singular ====<br /> <br /> Plurals must be used on objects when it's a ''One to Many'' relationship. Ex: bulk import, mass deletion, ... In any other case, use the singular.<br /> <br /> ==== Ends with a verb ====<br /> <br /> The last word (after the last underscore) must be a verb.<br /> <br /> === Deprecated events ===<br /> Following are the events that were supported in 2.5, but deprecated in 2.6 or later<br /> <br /> groups_groupings_deleted (see MDL-41312)<br /> <br /> groups_groupings_groups_removed (see MDL-41312)<br /> <br /> groups_groups_deleted (see MDL-41312)<br /> <br /> groups_members_removed (see MDL-41312)<br /> <br /> == Shared events ==<br /> <br /> Decision: Not supported at this stage.<br /> <br /> In Moodle 2.5 we have a good example of a shared event: 'assessable_content_uploaded' which is triggered in ''forum'', 'assignment'' and ''workshop''.<br /> <br /> The problem with shared events is that we cannot easily track what component triggered them. Of course we could add a new property to the event to keep track of that, but we would soon need more information and more properties. Also, in the case of a logger, the event received would be unique, where in fact it should be considered different depending on the component firing it.<br /> <br /> In our first implementation, we will create one specific event per module. This flexibility does not prevent any observer from capturing them, but still makes sure that the consistency and specificity of each event is maintained.<br /> <br /> It could happen that some events are defined in core and shared, but this should not really happen as low-level APIs should trigger the event, and a module should call that low API instead of doing the job itself.<br /> <br /> == One to many ==<br /> <br /> Decision: Each event should have a one to one relationship. We can reconsider this at a later stage, if the performance hit is extremely high.<br /> <br /> In 2.5, some events are triggered when an action happens on multiple objects. We have to decide whether we want to keep supporting ''One to Many'' events or not.<br /> <br /> Keeping a list of all changes for multiple actions may be problematic because you would have to keep them all in memory until all things are processed. This might also result in the order of events being incorrect. The only correct solution seems to be to trigger each item individually and then many things at the end. Performance needs to be improved elsewhere...<br /> <br /> === Accuracy ===<br /> <br /> When uploading a bunch of users using the CSV upload feature, if only one event is triggered, it means that the observers of ''user_created'' won't be triggered. And so some functionality can be lost as, as a plugin developer, I expect this ''user_created'' to be triggered regardless of the way they have been uploaded. Of course, the developer could observe the event ''bulk_user_imported'', but that means that he could miss some relevant observers.<br /> <br /> This applies to existing events.<br /> <br /> === Performance ===<br /> <br /> Triggering one event is cheaper then repeating the same events x number of times...<br /> <br /> === Information tracking ===<br /> <br /> A ''bulk'' event, might not be verbose enough to allow for proper logging afterwards. Though this is the responsibility of the logger, we probably want to make it easy to store relevant information.<br /> <br /> === Double event ===<br /> <br /> In the case of a bulk user import, if we were to trigger an event per user created, we probably want to trigger one event 'user_bulk_upload_started' when the action starts.<br /> <br /> == Unit Testing ==<br /> <br /> With unit testing for this system we want to assert the following:<br /> <br /> * That event strict validation and custom validation works.<br /> * Missing event data is auto filled with accurate data.<br /> * Typos in properties passed to ::create() are captured (if we decide to validate).<br /> * The legacy methods return the expected values (use assertEventLegacyData() and assertEventLegacyData())<br /> * The class properties are correctly overridden (crud, level, action, object, ...).<br /> * The properties automatically generated (component, name, ...) are correct.<br /> * Events are dispatched to the corresponding observers.<br /> * Events are dispatched to the corresponding legacy handlers.<br /> * Events are dispatched to the * observers.<br /> * Events perform an add_to_log() if it has legacy log data.<br /> * 'Events restore' restored the whole event data, and does not miss any information.<br /> * 'Events restore' does not generate any extra information.<br /> * Event methods should check context object or avoid using it, as context might not be valid at time of event restore (use assertEventContextNotUsed())<br /> <br /> = Example events =<br /> <br /> == Assignment ==<br /> <br /> Assumption: Course contains groups with students in each group.<br /> <br /> '''1.''' Teacher creates an assignment with group mode set to 'Separate groups' and Feedback type set to comments and files.<br /> *Event: User 'Teacher' has created assignment 'B' in course 'C101'.<br /> '''2.''' A student views the assignment.<br /> *Event: User 'Student' has viewed assignment 'B' in course 'C101'.<br /> '''3.''' A member from one of the groups submits an assignment<br /> *Event: User 'Student' has added a submission for assignment 'B' for group 'C' in course 'C101'.<br /> *Event: Email sent to the teacher and all students in that group.<br /> '''4.''' User 'Adrian' adds some changes to the assignment and updates it.<br /> *Event: User 'Adrian' has updated the submission for assignment 'B' for group 'C' in course 'C101'.<br /> *Event: Email sent to the teacher and all students in that group.<br /> '''5.''' Teacher views the assignment.<br /> *Event: User 'Teacher' has viewed assignment 'B' in course 'C101'.<br /> '''6.''' Teacher clicks on 'View/grade all submissions'<br /> *Event: User 'Teacher' has viewed the assignment 'B' grade area in course 'C101'.<br /> '''7.''' Teacher clicks to grade the student's submission.<br /> *Event: User 'Teacher' has viewed the submission for user 'student' for assignment 'B' in course 'C101'.<br /> '''8.''' Teacher marks the assignment with the setting 'Apply grades and feedback to entire group' set to 'Yes' leaving a comment and a file.<br /> *Event: User 'Teacher' has marked assignment 'B' for group 'C' in course 'C101'.<br /> *Event: User 'Teacher' has left a comment for assignment 'B' for group 'C' in course 'C101'.<br /> *Event: User 'Teacher' has uploaded a feedback file for assignment 'B' for group 'C' in course 'C101'.<br /> *Event: User 'Teacher' has uploaded a file to the course 'C101'.<br /> *Event: Email sent to user 'Student' notifying them their submission for assignment 'B' has been marked. - This is done for all users in the group.<br /> '''9.''' User 'Adrian' views the feedback.<br /> *Event: User 'Adrian' has viewed assignment 'B' in course 'C101'.<br /> '''10.''' User 'Adrian' opens the feedback file.<br /> *Event: User 'Adrian' has viewed the file 'A' for assignment 'B' in course 'C101'.<br /> '''11.''' User 'Adrian' adds some changes to the assignment insulting the teachers marking and updates it.<br /> *Event: User 'Adrian' has updated the submission for assignment 'B' for group 'C' in course 'C101'.<br /> *Event: Email sent to user 'Teacher' notifying them that user 'Adrian' has updated the submission for assignment 'B'.<br /> *Event: Email sent to user 'Student' notifying them their submission for assignment 'B' has been updated. - This is done for all users in the group.<br /> '''12.''' The teacher clicks directly on the link in the email to be taken to the grading page.<br /> *Event: User 'Teacher' has viewed the submission for user 'student' for assignment 'B' in course 'C101'.<br /> '''13.''' The teacher is upset due to the harsh comments and decides to mark Adrian down, but not the rest of the group by setting 'Apply grades and feedback to entire group' set to 'No'.<br /> *Event: User 'Teacher' has marked assignment 'B' for user 'Adrian' in course 'C101'.<br /> *Event: User 'Teacher' has left a comment for assignment 'B' for user 'Adrian' in course 'C101'.<br /> *Event: User 'Teacher' has uploaded a feedback file for assignment 'B' for user 'Adrian' in course 'C101'.<br /> *Event: Email sent to user 'Adrian' notifying them their submission for assignment 'B' has been marked.<br /> <br /> = FAQs =<br /> <br /> ; Why not use create events in core subsystems? : Because we could not see all core events in one place, it would create problems when naming event classes and finally subsystems are incomplete, we would have to add more now because we could not move the events in the future.<br /> <br /> = Development stages =<br /> <br /> == Stage 1 ==<br /> <br /> * Finish class loader spec and implement basic Frankenstyle class loader.<br /> * Describe new observer definition - just few new flags in current db/events.php<br /> * Describe new event dispatcher.<br /> * Describe core_event_base class.<br /> * Current (= legacy) events triggering:<br /> ** Re-factor current event handling code to new self-contained class - do not change functionality, keep events_trigger().<br /> ** Create new event handler management class that deals with installation and upgrades of both legacy and new handlers.<br /> * New events:<br /> ** Create new core_event_base class.<br /> ** Create new self-contained event dispatcher class with '*' handler support.<br /> ** In function core_event_base::trigger() check if the event has property 'legacyeventname' execute events_trigger($this-&gt;legacyeventname, $this-&gt;legacyeventdata) after triggering new event.<br /> ** Write unit tests for all new events code.<br /> * No changes to be made to the current logging system yet.<br /> <br /> After completing this stage everything should continue to work as it did before and we can start parallel work on further stages.<br /> <br /> == Stage 2 (requires completion of Stage 1) ==<br /> <br /> * Create event classes and replace existing calls to events_trigger() and with new event classes containing legacy information properties.<br /> * Add more events throughout the standard Moodle package in places where we have add_to_log(). Implement some_event::get_legacy_log_data() which returns original parameters of add_to_log() and remove it. Old add_to_log() function is called in core_event_base::trigger() automatically with original parameters.<br /> * Add even more new events all over the place.<br /> <br /> The difficult part is defining the new event classes properly because we must not change them after the 2.6 release.<br /> <br /> == Stage 3 (requires partial completion of Stage 2) ==<br /> <br /> * Migrate current legacy event handlers to new handlers with one event class instance parameter, ex.: enrol plugins.<br /> <br /> == Stage 4 (requires partial completion of Stage 2) ==<br /> <br /> * Implement an event logging handler.<br /> * Implement logging storage plugins.<br /> * Define logging apis.<br /> * Create new reports.<br /> * Switch to new logging everywhere after Stage 2 has been completed and new reports are usable.<br /> <br /> See [[Logging 2]]<br /> <br /> == Stage 5 ==<br /> <br /> * Decide how much backwards compatibility we want for old log tables. Most probably they will get only legacy log data.<br /> * Implement some BC solution for old code that reads log tables directly.<br /> <br /> See [[Logging 2]]<br /> <br /> == Stage 6 (requires completion of Stage 4 and 5) ==<br /> <br /> Moodle 2.8dev? This is the ultimate end of old logging via the ''log'' table.<br /> <br /> * Deprecate the add_to_log() function with a debug message and do nothing inside.<br /> * Remove all legacy logging from event classes.<br /> <br /> See [[Logging 2]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Projects_for_new_developers&diff=44005 2014-02-27T11:02:57Z

<p>Shashitechno: /* Event subscription plugin */</p> <hr /> <div>&lt;!--<br /> [[File:GSoC2012 logo.png|right]]<br /> <br /> &lt;p class=&quot;note&quot;&gt;'''Latest news:''' ''Moodle will be taking part in the [http://socghop.appspot.com/ Google Summer of Code] for our sixth year in 2012!''&lt;/p&gt;<br /> <br /> If you're thinking of applying to work with Moodle for GSOC 2012, please check the information on [[Applying to work with Moodle for GSOC]] and join our gsoc2012@conference.moodle.org jabber chat!<br /> --&gt;<br /> <br /> ==Getting started==<br /> <br /> Moodle uses PHP, JavaScript and a number of other Web languages, so learning those is a good place to start.<br /> <br /> When you have some basic PHP programming skills, you may wish to start learning about how to Moodle code is organised. It is recommended that you complete the [http://dev.moodle.org/course/view.php?id=2 Introduction to Moodle Programming] course on [http://dev.moodle.org/ dev.moodle.org]. To access this you will need to have an account on moodle.org first.<br /> <br /> ''If you are looking for projects suggested in the tracker, look for issues with the [https://tracker.moodle.org/issues/?jql=labels%20in%20%28addon_candidate%29 'addon_candidate' label].<br /> <br /> ''If you are looking to make a quick contribution, look for tracker issues with marked as [https://tracker.moodle.org/issues/?jql=Difficulty%20%3D%20Easy easy].<br /> <br /> As you become more involved in Moodle development, you might like to learn more about the [[Coding|coding conventions]] used and how changes to Moodle core code are [[Process|processed]].<br /> <br /> ==Potential projects==<br /> <br /> This evolving page lists possible Moodle projects for new developers derived from community suggestions.<br /> <br /> ''If you have any ideas for new features in Moodle which might be suitable as projects for new developers, please see [[New feature ideas]].''<br /> <br /> === SCORM ===<br /> <br /> There are various SCORM related projects I'd be interested in mentoring for GSOC in 2014. <br /> * Fix/Improve built in SCORM navigation (MDL-25642 MDL-29193 MDL-39551)<br /> * Improve SCORM performance, PHPDocs, compliance with coding guidelines. - There are a number of known areas of SCORM code that need a clean-up/re-structure - It would be good to audit all the scorm code and make it comply with Moodle's coding guidelines and improve the performance of the code - it would be good to implement MUC in areas that can make use of it - Ideally we would also rewrite the datamodels/scorm_*_.js.php files to separate the php from the js and make the JavaScript cache-able - (work started on this in MDL-35870) Other related bugs: MDL-41216 MDL-41665 MDL-42314 <br /> * SCORM reports - there are currently 3 SCORM reports - ideas on new report plugins would be considered as possible projects.<br /> <br /> :'''Skills required''': PHP, JavaScript<br /> :'''Difficulty level''': Medium -&gt; Hard<br /> :'''Possible mentor''': [http://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> <br /> ===HTML maths editor===<br /> <br /> Traditional written mathematical notation takes advantage of a rich set of special symbols, together with their relative size and<br /> position on a two dimensional page. Underlying mathematical expressions is a well-defined semantic tree structure. When typing a mathematical expression into a computer keyboard the ability to take advantage of the features of traditional mathematical notation is severely limited. Essentially one has only a one-dimensional string of symbols taken from the limited alphabet found on computer keyboards and a strict syntax. Syntax is often problematic for users. For example, they differ between applications and they do not correspond to traditional notation. <br /> <br /> DragMath (http://www.dragmath.bham.ac.uk) is a nice &quot;drag and drop&quot; equation editor that has been integrated with the Moodle editor for some time. However, it is implemented in Java, and that is becoming increasingly problematic, with increasing security warnings on the desktop, and limited support on mobile devices. We would like a new editor, keeping all the good parts of DragMath, but written in JavaScript. This could then be integrated with Moodle and other web applications.<br /> <br /> Goal: to create a useful and usable &quot;mathematics entry system&quot; for Moodle, using a combination of JavaScript and HTML5.<br /> <br /> This should<br /> * parse typed expressions into an internal tree representation<br /> * provide useful feedback to users, e.g. &quot;missing bracket&quot;, on ill-formed expressions.<br /> * have flexible options for providing a &quot;context&quot; to mediate between the requirements of a strict syntax, and users' expectations based on traditional written mathematics. E.g. is &quot;x(t+1)&quot; a multiplication of x and (t+1) or application of the function &quot;x&quot; to the argument &quot;(t+1)&quot;?<br /> * have &quot;drag and drop&quot; components (in HTML5)<br /> * have a modular and flexible output mechanism, this includes<br /> ** on-screen display &quot;as you type/edit&quot; <br /> ** output in a variety of formats, LaTeX, Maxima syntax, MathML etc. which can be embedded into web applications, specifically Moodle. It is not a goal to provide multiple outputs, but it is a goal to develop a framework in which other users can contribute such formats.<br /> ** show users the internal tree representation on request<br /> * potentially enable manipulation (computer algebra) of internal expressions by pre-defined rules.<br /> * have a well documented and simple API.<br /> <br /> Much of the basic design has been done in for example<br /> * DragMath http://www.dragmath.bham.ac.uk/<br /> * Numbas http://www.ncl.ac.uk/maths/numbas/<br /> * CanvasMath https://code.google.com/p/canvasmath/<br /> We will draw from this previous design experience to guide the development.<br /> <br /> See this forum thread: https://moodle.org/mod/forum/discuss.php?d=251627 for more.<br /> <br /> :'''Skills required''': JavaScript, HTML 5<br /> :'''Difficulty level''': Hard<br /> :'''Possible mentors''': Chris Sangwin<br /> <br /> ===New question types===<br /> <br /> This is not really a specific project idea, but I would like to point one an important general area:<br /> <br /> With HTML5 the range of what can be done in a web browser keeps expanding. Can we use these possibilities to make new, much more interactive, question types for Moodle? <br /> <br /> There are some ideas in this forum thread: https://moodle.org/mod/forum/discuss.php?d=222439<br /> * a question type where students have to join things up correctly by adding lines to a diagram (for example to complete an electric circuit).<br /> * a question type where students can change the colour of certain parts of a diagram, and they have to get it correct.<br /> * ... I am sure there are more possible ideas. Use your imagination!<br /> <br /> There are also some ideas which don't require complex HTML5 things:<br /> * A 'Give 3 examples of ...' Question type. For example &quot;Give three ways to speed up a chemical reaction:&quot;. Answer 'heat', 'increase concentration', 'catalyst' (in any order). However, 'warm it' might be an acceptable alternative, but 'heat', 'catalyst', 'warm it' should only score 2/3.<br /> * An ordering question type. Probably based on the OU's qtype_ddwtos. The think that cannot do is give good credit for partially correct answer. For example 'F', 'A', 'B', 'C', 'D', 'E' might be considered close to the right order, but if you do that with ddwtos it will score 0.<br /> * A question type where students must highlight certain words in some text. E.g. &quot;Find all the verbs in this paragraph.&quot;<br /> <br /> There is also scope to make significant enhancements to existing question types. For example<br /> * in [https://moodle.org/plugins/view.php?plugin=qtype_ddmarker Drag and drop markers] allow teachers to define the drop zones by dragging with the mouse, rather than typing co-ordinates.<br /> * in [https://moodle.org/plugins/view.php?plugin=qtype_pmatch the pattern-match question type] a tool like STACK question tests to help teachers verify that their answer matching works correctly (upload example responses, indicate what grade they should receive, show what grade they actually receive, and highlight the differences.<br /> <br /> Before proposing anything, please make sure you are familiar with the [https://docs.moodle.org/22/en/Questions standard question types] already available in Moodle, and the [http://moodle.org/plugins/browse.php?list=category&amp;id=29 contributed question types] that other people have already created.<br /> <br /> :'''Skills required''': PHP &amp; JavaScript<br /> :'''Difficulty level''': Medium - Hard<br /> :'''Possible mentors''': Tim Hunt<br /> <br /> === Plagiarism plugin ===<br /> <br /> There are various commercial plugins available that use the Plagiarism API in Moodle, but because these plagiarism systems can require paid subscriptions, testing them can be difficult. I'd like to see a basic plugin developed that could be used for testing the Plagiarism API, providing simple useful functionality. All Plagiarism API functions for all Modules that support the API should be implemented and used. It could check the file contenthash against all other files uploaded in the Moodle files table and if a matching record is found it should display information about the duplicate submission including if the same student has submitted the file in a different course/activity or if a different user has uploaded the file with the same contenthash.<br /> <br /> The first stage of this project is to implement very simple high performing code that allows easy testing of the Plagiarism API and provide some very basic functionality by checking the contenthash <br /> This code should implement unit tests and behat tests so that we can verify that the Moodle Plagiarism API is functioning correctly.<br /> <br /> The long term plan (possibly after GSOC) would be to get this plugin to post content to an external source which could implement a more complete plagiarism check running on a separate server so it doesn't affect performance of the Moodle site. The Plagiarism plugin &quot;Crot&quot; may have some code that could be repurposed to use for this.<br /> <br /> :'''Skills required''': PHP<br /> :'''Difficulty level''': Medium<br /> :'''Possible mentor''': [http://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> :'''Discussion''': [http://dev.moodle.org/mod/forum/discuss.php?d=1822]<br /> <br /> === Event subscription plugin ===<br /> <br /> This project includes the following:-<br /> * creating a Moodle plugin that provides users with an interface to subscribe to various events in Moodle.<br /> * Users should be able to subscribe to generic events (ex:- all delete events) or extremelly specfic events (ex:- users added to a specific group)<br /> * Performance is an important aspect, any solution designed should not be too harsh on performance.<br /> * This project requires deep understanding of Moodle events framework - [https://docs.moodle.org/dev/Event_2 Event Specs]<br /> <br /> :'''Skills required''': PHP and Javascript<br /> :'''Difficulty level''': Medium<br /> :'''Possible mentor''': [https://moodle.org/user/profile.php?id=1310725, Ankit Agarwal]<br /> <br /> ==See also==<br /> <br /> * [[GSOC]] - describing Moodle's involvement with Google in their Summer of Code program<br /> * [http://tracker.moodle.org/secure/IssueNavigator!executeAdvanced.jspa?jqlQuery=type+in+%28%22New+Feature%22%2C+Improvement%29+AND+resolution+%3D+unresolved+ORDER+BY+votes+DESC&amp;runQuery=true&amp;clear=true Popular new feature and improvement requests in Tracker]<br /> <br /> [[Category:GSOC]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Projects_for_new_developers&diff=44004 2014-02-27T11:02:11Z

<p>Shashitechno: /* Event subscription plugin */</p> <hr /> <div>&lt;!--<br /> [[File:GSoC2012 logo.png|right]]<br /> <br /> &lt;p class=&quot;note&quot;&gt;'''Latest news:''' ''Moodle will be taking part in the [http://socghop.appspot.com/ Google Summer of Code] for our sixth year in 2012!''&lt;/p&gt;<br /> <br /> If you're thinking of applying to work with Moodle for GSOC 2012, please check the information on [[Applying to work with Moodle for GSOC]] and join our gsoc2012@conference.moodle.org jabber chat!<br /> --&gt;<br /> <br /> ==Getting started==<br /> <br /> Moodle uses PHP, JavaScript and a number of other Web languages, so learning those is a good place to start.<br /> <br /> When you have some basic PHP programming skills, you may wish to start learning about how to Moodle code is organised. It is recommended that you complete the [http://dev.moodle.org/course/view.php?id=2 Introduction to Moodle Programming] course on [http://dev.moodle.org/ dev.moodle.org]. To access this you will need to have an account on moodle.org first.<br /> <br /> ''If you are looking for projects suggested in the tracker, look for issues with the [https://tracker.moodle.org/issues/?jql=labels%20in%20%28addon_candidate%29 'addon_candidate' label].<br /> <br /> ''If you are looking to make a quick contribution, look for tracker issues with marked as [https://tracker.moodle.org/issues/?jql=Difficulty%20%3D%20Easy easy].<br /> <br /> As you become more involved in Moodle development, you might like to learn more about the [[Coding|coding conventions]] used and how changes to Moodle core code are [[Process|processed]].<br /> <br /> ==Potential projects==<br /> <br /> This evolving page lists possible Moodle projects for new developers derived from community suggestions.<br /> <br /> ''If you have any ideas for new features in Moodle which might be suitable as projects for new developers, please see [[New feature ideas]].''<br /> <br /> === SCORM ===<br /> <br /> There are various SCORM related projects I'd be interested in mentoring for GSOC in 2014. <br /> * Fix/Improve built in SCORM navigation (MDL-25642 MDL-29193 MDL-39551)<br /> * Improve SCORM performance, PHPDocs, compliance with coding guidelines. - There are a number of known areas of SCORM code that need a clean-up/re-structure - It would be good to audit all the scorm code and make it comply with Moodle's coding guidelines and improve the performance of the code - it would be good to implement MUC in areas that can make use of it - Ideally we would also rewrite the datamodels/scorm_*_.js.php files to separate the php from the js and make the JavaScript cache-able - (work started on this in MDL-35870) Other related bugs: MDL-41216 MDL-41665 MDL-42314 <br /> * SCORM reports - there are currently 3 SCORM reports - ideas on new report plugins would be considered as possible projects.<br /> <br /> :'''Skills required''': PHP, JavaScript<br /> :'''Difficulty level''': Medium -&gt; Hard<br /> :'''Possible mentor''': [http://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> <br /> ===HTML maths editor===<br /> <br /> Traditional written mathematical notation takes advantage of a rich set of special symbols, together with their relative size and<br /> position on a two dimensional page. Underlying mathematical expressions is a well-defined semantic tree structure. When typing a mathematical expression into a computer keyboard the ability to take advantage of the features of traditional mathematical notation is severely limited. Essentially one has only a one-dimensional string of symbols taken from the limited alphabet found on computer keyboards and a strict syntax. Syntax is often problematic for users. For example, they differ between applications and they do not correspond to traditional notation. <br /> <br /> DragMath (http://www.dragmath.bham.ac.uk) is a nice &quot;drag and drop&quot; equation editor that has been integrated with the Moodle editor for some time. However, it is implemented in Java, and that is becoming increasingly problematic, with increasing security warnings on the desktop, and limited support on mobile devices. We would like a new editor, keeping all the good parts of DragMath, but written in JavaScript. This could then be integrated with Moodle and other web applications.<br /> <br /> Goal: to create a useful and usable &quot;mathematics entry system&quot; for Moodle, using a combination of JavaScript and HTML5.<br /> <br /> This should<br /> * parse typed expressions into an internal tree representation<br /> * provide useful feedback to users, e.g. &quot;missing bracket&quot;, on ill-formed expressions.<br /> * have flexible options for providing a &quot;context&quot; to mediate between the requirements of a strict syntax, and users' expectations based on traditional written mathematics. E.g. is &quot;x(t+1)&quot; a multiplication of x and (t+1) or application of the function &quot;x&quot; to the argument &quot;(t+1)&quot;?<br /> * have &quot;drag and drop&quot; components (in HTML5)<br /> * have a modular and flexible output mechanism, this includes<br /> ** on-screen display &quot;as you type/edit&quot; <br /> ** output in a variety of formats, LaTeX, Maxima syntax, MathML etc. which can be embedded into web applications, specifically Moodle. It is not a goal to provide multiple outputs, but it is a goal to develop a framework in which other users can contribute such formats.<br /> ** show users the internal tree representation on request<br /> * potentially enable manipulation (computer algebra) of internal expressions by pre-defined rules.<br /> * have a well documented and simple API.<br /> <br /> Much of the basic design has been done in for example<br /> * DragMath http://www.dragmath.bham.ac.uk/<br /> * Numbas http://www.ncl.ac.uk/maths/numbas/<br /> * CanvasMath https://code.google.com/p/canvasmath/<br /> We will draw from this previous design experience to guide the development.<br /> <br /> See this forum thread: https://moodle.org/mod/forum/discuss.php?d=251627 for more.<br /> <br /> :'''Skills required''': JavaScript, HTML 5<br /> :'''Difficulty level''': Hard<br /> :'''Possible mentors''': Chris Sangwin<br /> <br /> ===New question types===<br /> <br /> This is not really a specific project idea, but I would like to point one an important general area:<br /> <br /> With HTML5 the range of what can be done in a web browser keeps expanding. Can we use these possibilities to make new, much more interactive, question types for Moodle? <br /> <br /> There are some ideas in this forum thread: https://moodle.org/mod/forum/discuss.php?d=222439<br /> * a question type where students have to join things up correctly by adding lines to a diagram (for example to complete an electric circuit).<br /> * a question type where students can change the colour of certain parts of a diagram, and they have to get it correct.<br /> * ... I am sure there are more possible ideas. Use your imagination!<br /> <br /> There are also some ideas which don't require complex HTML5 things:<br /> * A 'Give 3 examples of ...' Question type. For example &quot;Give three ways to speed up a chemical reaction:&quot;. Answer 'heat', 'increase concentration', 'catalyst' (in any order). However, 'warm it' might be an acceptable alternative, but 'heat', 'catalyst', 'warm it' should only score 2/3.<br /> * An ordering question type. Probably based on the OU's qtype_ddwtos. The think that cannot do is give good credit for partially correct answer. For example 'F', 'A', 'B', 'C', 'D', 'E' might be considered close to the right order, but if you do that with ddwtos it will score 0.<br /> * A question type where students must highlight certain words in some text. E.g. &quot;Find all the verbs in this paragraph.&quot;<br /> <br /> There is also scope to make significant enhancements to existing question types. For example<br /> * in [https://moodle.org/plugins/view.php?plugin=qtype_ddmarker Drag and drop markers] allow teachers to define the drop zones by dragging with the mouse, rather than typing co-ordinates.<br /> * in [https://moodle.org/plugins/view.php?plugin=qtype_pmatch the pattern-match question type] a tool like STACK question tests to help teachers verify that their answer matching works correctly (upload example responses, indicate what grade they should receive, show what grade they actually receive, and highlight the differences.<br /> <br /> Before proposing anything, please make sure you are familiar with the [https://docs.moodle.org/22/en/Questions standard question types] already available in Moodle, and the [http://moodle.org/plugins/browse.php?list=category&amp;id=29 contributed question types] that other people have already created.<br /> <br /> :'''Skills required''': PHP &amp; JavaScript<br /> :'''Difficulty level''': Medium - Hard<br /> :'''Possible mentors''': Tim Hunt<br /> <br /> === Plagiarism plugin ===<br /> <br /> There are various commercial plugins available that use the Plagiarism API in Moodle, but because these plagiarism systems can require paid subscriptions, testing them can be difficult. I'd like to see a basic plugin developed that could be used for testing the Plagiarism API, providing simple useful functionality. All Plagiarism API functions for all Modules that support the API should be implemented and used. It could check the file contenthash against all other files uploaded in the Moodle files table and if a matching record is found it should display information about the duplicate submission including if the same student has submitted the file in a different course/activity or if a different user has uploaded the file with the same contenthash.<br /> <br /> The first stage of this project is to implement very simple high performing code that allows easy testing of the Plagiarism API and provide some very basic functionality by checking the contenthash <br /> This code should implement unit tests and behat tests so that we can verify that the Moodle Plagiarism API is functioning correctly.<br /> <br /> The long term plan (possibly after GSOC) would be to get this plugin to post content to an external source which could implement a more complete plagiarism check running on a separate server so it doesn't affect performance of the Moodle site. The Plagiarism plugin &quot;Crot&quot; may have some code that could be repurposed to use for this.<br /> <br /> :'''Skills required''': PHP<br /> :'''Difficulty level''': Medium<br /> :'''Possible mentor''': [http://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> :'''Discussion''': [http://dev.moodle.org/mod/forum/discuss.php?d=1822]<br /> <br /> === Event subscription plugin ===<br /> <br /> This project includes the following:-<br /> * creating a Moodle plugin that provides users with an interface to subscribe to various events in Moodle.<br /> * Users should be able to subscribe to generic events (ex:- all delete events) or extremelly specfic events (ex:- users added to a specific group)<br /> * Performance is an important aspect, any solution designed should not be too harsh on performance.<br /> * This project requires deep understanding of Moodle events framework - [https://docs.moodle.org/dev/Event_2, Event Specs]<br /> <br /> :'''Skills required''': PHP and Javascript<br /> :'''Difficulty level''': Medium<br /> :'''Possible mentor''': [https://moodle.org/user/profile.php?id=1310725, Ankit Agarwal]<br /> <br /> ==See also==<br /> <br /> * [[GSOC]] - describing Moodle's involvement with Google in their Summer of Code program<br /> * [http://tracker.moodle.org/secure/IssueNavigator!executeAdvanced.jspa?jqlQuery=type+in+%28%22New+Feature%22%2C+Improvement%29+AND+resolution+%3D+unresolved+ORDER+BY+votes+DESC&amp;runQuery=true&amp;clear=true Popular new feature and improvement requests in Tracker]<br /> <br /> [[Category:GSOC]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42344 2013-09-19T13:21:18Z

<p>Shashitechno: /* Installation */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * works consistently on different DB engines and different content languages( Non latin languages support)<br /> <br /> <br /> * Search results with relevance( Score )<br /> <br /> <br /> * Case insensitive search capability<br /> <br /> <br /> * Searchable Document Formats<br /> <br /> <br /> * Hyper Text Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * Open Document Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Autocomplete ( autosuggest with non-latin languages too )<br /> <br /> <br /> * Very fast than existing course search implementation. <br /> <br /> <br /> * Spell Checking capability ( Did you mean ?)<br /> <br /> <br /> * Keyword Matching (Searching within specific field)<br /> <br /> <br /> Example :- &quot;summary:Getting started with python&quot; searches for the &quot;Getting started with python&quot; in summary field of courses.<br /> <br /> <br /> * Proximity Search<br /> <br /> Example :- &quot;moodle perth&quot;~4 Search for &quot;moodle perth&quot; within 4 words from each other.<br /> <br /> <br /> * Wildcard Search<br /> <br /> <br /> * Fuzzy search support ( Alternate form of words )<br /> <br /> <br /> * Filtering results by startdate (Range queries)<br /> <br /> <br /> * Pagination &amp; sorting results by relevance, startdate, shortname.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> === Prerequisite ===<br /> <br /> Java 5 or higher (a.k.a. 1.5.x), PHP 5.1.4 or higher, moodle 2.5 or higher.<br /> <br /> === Installing admin tool ===<br /> <br /> 1. Download the admin tool from here. (https://moodle.org/plugins/view.php?plugin=tool_coursesearch) <br /> <br /> 2. Extract the Course Search folder. Put it under moodle installation/admin/tool directory.<br /> <br /> 3. It should be named coursesearch. If you are already logged in just refreshing the browser should trigger your moodle <br /> site to begin the install 'Plugins Check'.<br /> <br /> 4. If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme OR replacing/copying the renderer.php file ===<br /> <br /> <br /> Here is two options either you may replace/copy your existing theme renderer with the one found in<br /> search_cleantheme(https://github.com/shashirepo/moodle-theme_cleantheme).<br /> OR you may install cleantheme itself.<br /> <br /> replacing /copying the renderer file to your theme.<br /> <br /> 1. Copy renderer.php file from search_cleantheme replace it with your theme's renderer file.<br /> <br /> 2. Standard moodle theme doesn't have any renderer file so you may simply copy this to your<br /> theme directory.<br /> <br /> 3. Rename renderer class name according to your theme name.<br /> <br /> for example if you are using theme 'clean'.<br /> then rename the class names to 'theme_clean_core_renderer' &amp; 'theme_clean_core_course_renderer'.<br /> <br /> OR you may either use search_cleantheme. This is based on bootstrap clean theme.<br /> <br /> 1. Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme) <br /> <br /> 2. Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> 3. If you are already logged in just refreshing the browser should trigger your Moodle<br /> site to begin the install 'Plugins Check'.<br /> <br /> 4. If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> <br /> Download the latest Solr 4.4.0 release from:<br /> http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot<br /> and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for<br /> testing, development, and even for smaller production sites. This<br /> application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to<br /> something like schema.bak. Then move the schema.xml that comes <br /> with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename<br /> it like solrconfig.bak. Then move the solrconfig.xml that comes with the<br /> moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename<br /> it like protwords.bak. Then move the protwords.txt that comes with the<br /> moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core<br /> may not load if you don't have at least an empty file present:<br /> solrconfig.xml<br /> schema.xml<br /> elevate.xml<br /> mapping-ISOLatin1Accent.txt<br /> protwords.txt<br /> stopwords.txt<br /> synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to<br /> apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> Test that your solr server is now available by visiting<br /> http://localhost:8983/solr/admin/<br /> <br /> <br /> === Testing with ping to solr ===<br /> <br /> 1. Advance Course can be found under :-<br /> <br /> Administration-&gt;course-&gt;Course search settings(URL:- http://127.0.0.1/MoodleInstalltionURL/admin/tool/coursesearch)<br /> <br /> 2. Give the solr configuration options here:-<br /> <br /> Solr Host:- localhost or 127.0.0.1<br /> Solr Port:- 8983 (Default port for Solr )<br /> Solr path :- /solr (Configuration directory for solr)<br /> <br /> 3. Click on &quot;Check Solr instance Setting&quot;. if it Shows ping successful(with an success image). Now click save changes.<br /> <br /> 4. Now click on &quot;Index courses&quot; to index all the courses. After successful indexing. It will come up with a success image.<br /> <br /> 5. Click on &quot;Optimize&quot; to optimize the existing indexes And improve solr performance.<br /> <br /> <br /> Enjoy the Search by going on page (http://127.0.0.1/MoodleInstallationURL/course/search.php)<br /> <br /> To configure navigation with the course search page goto Site administration -&gt; Front page -&gt; Front page settings.<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42343 2013-09-19T13:20:00Z

<p>Shashitechno: /* Features */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * works consistently on different DB engines and different content languages( Non latin languages support)<br /> <br /> <br /> * Search results with relevance( Score )<br /> <br /> <br /> * Case insensitive search capability<br /> <br /> <br /> * Searchable Document Formats<br /> <br /> <br /> * Hyper Text Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * Open Document Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Autocomplete ( autosuggest with non-latin languages too )<br /> <br /> <br /> * Very fast than existing course search implementation. <br /> <br /> <br /> * Spell Checking capability ( Did you mean ?)<br /> <br /> <br /> * Keyword Matching (Searching within specific field)<br /> <br /> <br /> Example :- &quot;summary:Getting started with python&quot; searches for the &quot;Getting started with python&quot; in summary field of courses.<br /> <br /> <br /> * Proximity Search<br /> <br /> Example :- &quot;moodle perth&quot;~4 Search for &quot;moodle perth&quot; within 4 words from each other.<br /> <br /> <br /> * Wildcard Search<br /> <br /> <br /> * Fuzzy search support ( Alternate form of words )<br /> <br /> <br /> * Filtering results by startdate (Range queries)<br /> <br /> <br /> * Pagination &amp; sorting results by relevance, startdate, shortname.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42242 2013-08-29T06:34:36Z

<p>Shashitechno: /* Schedule */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42241 2013-08-29T06:33:31Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|700px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42240 2013-08-29T06:32:33Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|700px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42239 2013-08-29T06:31:05Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|700px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|700px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42238 2013-08-29T06:29:49Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42237 2013-08-29T06:29:14Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42236 2013-08-29T06:28:13Z

<p>Shashitechno: /* Schedule */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42235 2013-08-29T06:27:32Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42234 2013-08-29T06:25:40Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=42233 2013-08-29T06:24:16Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''*Advance course search auto-suggestions''' <br /> <br /> [[File:autocomplete.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=File:autocomplete.png&diff=42232 2013-08-29T06:21:00Z

<p>Shashitechno: Moodle advance course search plugin autocomplete feature.</p> <hr /> <div>Moodle advance course search plugin autocomplete feature.</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41874 2013-08-11T18:04:00Z

<p>Shashitechno: /* See also */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> <br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41873 2013-08-11T17:56:19Z

<p>Shashitechno: /* Requirements */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> Require moodle 2.5 and Apache solr 4.x<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41872 2013-08-11T17:31:55Z

<p>Shashitechno: /* Features */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41849 2013-08-08T19:14:33Z

<p>Shashitechno: /* Schedule */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41848 2013-08-08T19:13:54Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41847 2013-08-08T19:13:09Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41846 2013-08-08T19:12:30Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41845 2013-08-08T19:11:16Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41844 2013-08-08T19:10:26Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> <br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41843 2013-08-08T19:09:17Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> <br /> <br /> [[File:Course_search_Adminui.png|1000px|thumb|left]]<br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> <br /> <br /> <br /> [[File:Course_Search_IntraWordQuery.png|1000px|thumb|left]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41842 2013-08-08T19:00:46Z

<p>Shashitechno: /* Installing Solr &amp; placing the plugin Schema */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: <br /> * solrconfig.xml <br /> * schema.xml<br /> * elevate.xml<br /> * mapping-ISOLatin1Accent.txt<br /> * protwords.txt<br /> * stopwords.txt<br /> * synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png]]<br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> <br /> [[File:Course_Search_IntraWordQuery.png]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41841 2013-08-08T18:57:57Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: solrconfig.xml schema.xml elevate.xml mapping-ISOLatin1Accent.txt protwords.txt stopwords.txt synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png]]<br /> <br /> <br /> <br /> <br /> '''* Course search results''' <br /> <br /> <br /> [[File:Course_Search_IntraWordQuery.png]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=File:Course_Search_IntraWordQuery.png&diff=41840 2013-08-08T18:56:44Z

<p>Shashitechno: plugin Course search Results</p> <hr /> <div>plugin Course search Results</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41839 2013-08-08T18:55:06Z

<p>Shashitechno: /* Design */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: solrconfig.xml schema.xml elevate.xml mapping-ISOLatin1Accent.txt protwords.txt stopwords.txt synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Course search plugin snapshots:-'''<br /> <br /> '''* Course search admin tool'''<br /> <br /> [[File:Course_search_Adminui.png]]<br /> <br /> <br /> '''* Course search results''' <br /> <br /> <br /> [[File:Course_search_Adminui.png]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=File:Course_search_Adminui.png&diff=41838 2013-08-08T18:52:05Z

<p>Shashitechno: Course Search plugin Admin UI</p> <hr /> <div>Course Search plugin Admin UI</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41837 2013-08-08T18:48:20Z

<p>Shashitechno: /* Features */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non Latin support with search queries.<br /> <br /> * Search results with relevance(Score)<br /> <br /> * Searchable Document Formats<br /> <br /> * HyperText Markup Language<br /> * XML and derived formats<br /> * Microsoft Office document formats<br /> * OpenDocument Format<br /> * Portable Document Format <br /> * Rich Text Format<br /> * Compression and packaging formats [See Rebuilding Solr Cell]<br /> * Text formats<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> * Sorting results by score, by shortname, by startdate<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: solrconfig.xml schema.xml elevate.xml mapping-ISOLatin1Accent.txt protwords.txt stopwords.txt synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41836 2013-08-08T18:44:08Z

<p>Shashitechno: /* Installing Solr &amp; placing the plugin Schema = */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ===<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: solrconfig.xml schema.xml elevate.xml mapping-ISOLatin1Accent.txt protwords.txt stopwords.txt synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41835 2013-08-08T18:43:24Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Installation ==<br /> <br /> Setting up Advance course search is quite easy ! you just need to go thorough these steps.<br /> <br /> === Installing Admin tool ===<br /> <br /> * Download the admin tool from here (https://github.com/shashirepo/moodle-tool_coursesearch)<br /> <br /> * Extract the Course Search folder. and put it under moodle installation /admin/tool directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications.<br /> <br /> === Installing search_cleantheme ===<br /> <br /> * Download the cleantheme from here (https://github.com/shashirepo/moodle-theme_cleantheme)<br /> <br /> * Extract the theme folder. and put it under moodle installation theme directory.<br /> <br /> * If you are already logged in just refreshing the browser should trigger your Moodle site to begin the install 'Plugins Check'.<br /> <br /> * If not then navigate to Administration &gt; Notifications. <br /> <br /> === Installing Solr &amp; placing the plugin Schema ====<br /> <br /> Download the latest Solr 4.4.0 release from: http://lucene.apache.org/solr/<br /> <br /> Unpack the tarball somewhere not visible to the web (not in your apache docroot and not inside of your moodle directory).<br /> <br /> The Solr download comes with an example application that you can use for testing, development, and even for smaller production sites. This application is found at apache-solr-4.4.0/example.<br /> <br /> Move apache-solr-4.4.0/example/solr/collection1/conf/schema.xml and rename it to something like schema.bak. Then move the schema.xml that comes with moodle course search admin tool plugin to take its place.<br /> <br /> Similarly, move apache-solr-4.4.0/example/solr/collection1/conf/solrconfig.xml and rename it like solrconfig.bak. Then move the solrconfig.xml that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Finally, move apache-solr-4.4.0/example/solr/collection1/conf/protwords.txt and rename it like protwords.bak. Then move the protwords.txt that comes with the moodle course search admin tool plugin to take its place.<br /> <br /> Make sure that the conf directory includes the following files - the Solr core may not load if you don't have at least an empty file present: solrconfig.xml schema.xml elevate.xml mapping-ISOLatin1Accent.txt protwords.txt stopwords.txt synonyms.txt<br /> <br /> Now start the solr application by opening a shell, changing directory to apache-solr-4.4.0/example, and executing the command java -jar start.jar<br /> <br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> <br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=41434 2013-07-18T20:51:39Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> [https://moodle.org/mod/forum/discuss.php?d=232803 Course Search plugin Progress]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40989 2013-06-19T09:39:47Z

<p>Shashitechno: /* Requirements */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.5.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40880 2013-06-12T08:54:32Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = [https://moodle.org/mod/forum/discuss.php?d=230469 Writing Moodle's Course Search plugin]<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=GSOC/2013&diff=40829 2013-06-09T18:11:14Z

<p>Shashitechno: /* Course search */</p> <hr /> <div>[[Image:banner-gsoc2013.png|right|400px]]<br /> <br /> [[GSOC|GSoC]] &gt; GSoC 2013<br /> <br /> An overview of the [http://www.google-melange.com/gsoc/program/home/google/gsoc2013 Google Summer of Code 2013] projects for Moodle..<br /> <br /> ==Self-assessment activity using the question bank==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1580475&amp;course=5 Jayesh Anandani]<br /> :'''Mentors''': [https://moodle.org/user/profile.php?id=8026&amp;course=5 Jean-Michel Vedrine] &amp; [https://moodle.org/user/view.php?id=93821&amp;course=5 Tim Hunt]<br /> :'''Specification''': [[Self-assessment activity using the question bank]]<br /> See also:<br /> * CONTRIB-4334<br /> * Developer blog link coming<br /> <br /> ==SCORM player rewrite==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1277285&amp;course=5 Mayank Gupta]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> :'''Specification''': [[SCORM player rewrite]]<br /> See also:<br /> * MDL-39910<br /> * Developer blog link coming<br /> <br /> ==Global search==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1565904&amp;course=5 Prateek Sachan]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=353008&amp;course=5 Tomasz Muras] &amp; [https://moodle.org/user/view.php?id=1146834&amp;course=5 Aparup Banerjee] <br /> :'''Specification''': [[Global search]]<br /> See also:<br /> * MDL-31989<br /> * Developer blog link coming<br /> <br /> ==Course search==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> :'''Mentor''': [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/view.php?id=381842&amp;course=5 Michael de Raadt]<br /> :'''Specification''': [[Course search]]<br /> See also:<br /> * CONTRIB-4335<br /> * Developer blog [http://shashitechno.wordpress.com/ # Shashikant Vaishnav]<br /> <br /> ==Pronunciation evaluation question type==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1594272 Troy Lee]<br /> :'''Mentors''': [https://moodle.org/user/view.php?id=1081865&amp;course=5 James Salsman] &amp; [https://moodle.org/user/view.php?id=93821&amp;course=5 Tim Hunt]<br /> :'''Specification''': [[Pronunciation evaluation question type]]<br /> See also:<br /> * CONTRIB-4336<br /> * [http://pronunciationeval.blogspot.sg/ Developer blog]<br /> <br /> ==BioAuth: A Moodle plugin for determining Quiz authorship==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1595676 Vinnie Monaco]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> :'''Specification''': [[BioAuth Plugin]]<br /> See also:<br /> * CONTRIB-4337<br /> * [http://vmonaco.com/blog Developer blog]<br /> <br /> ==Portfolio plugin for Evernote==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1459982 Vishal Raheja]<br /> :'''Mentors''': [https://moodle.org/user/profile.php?id=1451616 Frédéric Massart] &amp; [https://moodle.org/user/view.php?id=381842&amp;course=5 Michael de Raadt]<br /> :'''Specification''': [[Portfolio plugin for Evernote]]<br /> See also:<br /> * CONTRIB-4338<br /> * Developer blog link coming<br /> <br /> [[Category:GSOC]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=GSOC/2013&diff=40828 2013-06-09T18:10:40Z

<p>Shashitechno: /* Course search */</p> <hr /> <div>[[Image:banner-gsoc2013.png|right|400px]]<br /> <br /> [[GSOC|GSoC]] &gt; GSoC 2013<br /> <br /> An overview of the [http://www.google-melange.com/gsoc/program/home/google/gsoc2013 Google Summer of Code 2013] projects for Moodle..<br /> <br /> ==Self-assessment activity using the question bank==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1580475&amp;course=5 Jayesh Anandani]<br /> :'''Mentors''': [https://moodle.org/user/profile.php?id=8026&amp;course=5 Jean-Michel Vedrine] &amp; [https://moodle.org/user/view.php?id=93821&amp;course=5 Tim Hunt]<br /> :'''Specification''': [[Self-assessment activity using the question bank]]<br /> See also:<br /> * CONTRIB-4334<br /> * Developer blog link coming<br /> <br /> ==SCORM player rewrite==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1277285&amp;course=5 Mayank Gupta]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> :'''Specification''': [[SCORM player rewrite]]<br /> See also:<br /> * MDL-39910<br /> * Developer blog link coming<br /> <br /> ==Global search==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1565904&amp;course=5 Prateek Sachan]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=353008&amp;course=5 Tomasz Muras] &amp; [https://moodle.org/user/view.php?id=1146834&amp;course=5 Aparup Banerjee] <br /> :'''Specification''': [[Global search]]<br /> See also:<br /> * MDL-31989<br /> * Developer blog link coming<br /> <br /> ==Course search==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> :'''Mentor''': [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/view.php?id=381842&amp;course=5 Michael de Raadt]<br /> :'''Specification''': [[Course search]]<br /> See also:<br /> * CONTRIB-4335<br /> * Developer blog [http://shashitechno.wordpress.com/ #Shashikant Vaishnav]<br /> <br /> ==Pronunciation evaluation question type==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1594272 Troy Lee]<br /> :'''Mentors''': [https://moodle.org/user/view.php?id=1081865&amp;course=5 James Salsman] &amp; [https://moodle.org/user/view.php?id=93821&amp;course=5 Tim Hunt]<br /> :'''Specification''': [[Pronunciation evaluation question type]]<br /> See also:<br /> * CONTRIB-4336<br /> * [http://pronunciationeval.blogspot.sg/ Developer blog]<br /> <br /> ==BioAuth: A Moodle plugin for determining Quiz authorship==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1595676 Vinnie Monaco]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> :'''Specification''': [[BioAuth Plugin]]<br /> See also:<br /> * CONTRIB-4337<br /> * [http://vmonaco.com/blog Developer blog]<br /> <br /> ==Portfolio plugin for Evernote==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1459982 Vishal Raheja]<br /> :'''Mentors''': [https://moodle.org/user/profile.php?id=1451616 Frédéric Massart] &amp; [https://moodle.org/user/view.php?id=381842&amp;course=5 Michael de Raadt]<br /> :'''Specification''': [[Portfolio plugin for Evernote]]<br /> See also:<br /> * CONTRIB-4338<br /> * Developer blog link coming<br /> <br /> [[Category:GSOC]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40827 2013-06-09T17:45:52Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> Earlier we had two choices for this purpose - Apache Solr and Sphinx. Both Apache solr and Sphinx has extensive documentation and community support and have very powerful indexing capabilities, but eventually faceted search and fuzzy search features that works out of the box with some other key points explained below for Apache Solr made it the winning solution.<br /> <br /> === Why Apache Solr ===<br /> <br /> * As in Courses we are going to deal with Different types of summary documents &amp; Solr can index proprietary formats like Microsoft Word, Pdf, etc. Sphinx can't.<br /> <br /> * Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.<br /> <br /> * As we were concerned about the non-latin character support, Solr supports unicode character search out of the box.<br /> <br /> * Solr, being an Apache project, is obviously Apache2-licensed. So the real challenge is now to integrate Apache Solr with moodle course search functionality. Advance course search will be installed as a plugin to substitute basic core search.<br /> <br /> <br /> === Project Roadmap ===<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40826 2013-06-09T17:38:48Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Problem Description ==<br /> <br /> '''Non latin support with search queries:''' Database engines don't recognize word boundaries in non-english languages and can't do case-insensitive search.<br /> <br /> <br /> '''Indexing:''' We don't want to reinvent the wheel as there are many awesome open source enterprise level search indexers available that can make our course search fast and efficient. <br /> <br /> <br /> '''Sorting by relevance:''' We need to sort results according to relevance for example if a query result matches course name then it's more relevant then its other matches with summary of course. Implementing spelling correction (Did you mean?) feature. Also fuzzy search (alternate form of words) needs to be implemented.<br /> <br /> <br /> '''Work consistently on different database engines and different content language:''' Course search should be implemented in a way that it can deal with as many databases possible. and it should be able to deal with different content language.<br /> <br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40825 2013-06-09T17:34:40Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40824 2013-06-09T17:34:08Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Implementation ==<br /> <br /> * '''Preparing course search schema to integrate with Apache solr :''' Solr reads the schema to be indexed using a xml based configuration file, lets call it schema.xml. When user will install advance search plugin, it will contain its schema file that will be automatically replaced by the corresponding files located on the Solr server, and it can usually be found in the main Solr directory solr/conf. We will need to design moodle course search schema.xml depending on what fields we want to index and how. Available field types and filter options in Solr makes it very flexible how we want to do that. Fortunately similar problem has already been sorted out by Drupal folks and I am quite familiar with that so here we can use this knowledge reference. Searchable fields in course are idnumber, shortname, fullname, summary, summary. We will start with these fields for designing the schema for moodle course search.<br /> <br /> <br /> * '''Extending apache Solr integration library with php to work with moodle :''' It is the Solr Php Client library that will talk between moodle and Solr. Here we need to extend these files to make it work with moodle course search plugin.<br /> <br /> <br /> * '''Indexing the course search data:''' The entry point of indexing courses is the Cron task. Whenever the cron will run, an associated callback will trigger the indexing procedure. The plugin will look for courses which are marked for indexing, then for each course an object that contains all the relevant field contents to be indexed, will be individually converted into XML query and sent to the Solr server for indexing. We can also create a custom event to notify other entities that we just performed an indexing operation. How much courses can be indexed at once can be an admin setting. On admin side we will also have means to mark a course as a candidate for reindexing, so -<br /> <br /> * We need to perform re-indexing(by Cron listening)<br /> - If a new coursed is created. <br /> <br /> - If a existing course is deleted or updated.<br /> <br /> - Or it is marked by admin to index again<br /> <br /> <br /> * '''Plugin search API functions :''' Internal search API functions will have these objectives -<br /> <br /> 1. Getting the basic search query via form submission<br /> <br /> 2. Translation of this search query into appropriate Apache Solr query and adding appropriate parameters for extra search options<br /> <br /> 3. Trigger the search by sending this query to locally running Apache Solr instance<br /> <br /> 4. Getting the search response and rendering the search results via proper theme function.<br /> <br /> As we know Advance search is plugin type code base so when installed it will overtake the responsibility of search system. So when the user activates advance search plugin, then moodle will interact with its internal search and to interact with apache solr in backend. These search functions will be responsible to 'generate' the required apache solr query and to speak with solr running instance.<br /> <br /> '''On caching -''' While retrieving the list of courses that match search string Apache Solr has the mechanism to cache repetitious search queries. It has two built-in in-memory caches. one for search results and one for documents. They are typically limited to several thousands entries each.<br /> <br /> Finally we will prepare the list of courses to show to the user, with filtering out hidden courses that user can not see, retrieve all additional information such as category name, course contacts, etc.<br /> <br /> <br /> * '''Achieving relevancy by using Solr's awesome faceted search feature:''' We can achieve relevancy by dynamic filtering results like searching start date(by days,months,years etc.) or spelling correct feature (Did you mean?) and in back-end Solr supports that out of box.<br /> <br /> Relevancy is made using the ''boost'' clause on the field you want to give higher relevance. These &quot;boost&quot; clause will be added on search fields by internal search API functions as per the need while generating the search query for Apache Solr. From the Solr docs I found these key points to achieve for relevant results.[2]<br /> <br /> By default Apache solr sorts given search results in order of relevancy, and relevancy is defined in terms of how much 'boost' we have given to particular field and if some field has to remain more relevant by default than other Apache Solr by altering the schema file.<br /> <br /> <br /> * '''Solr with alternate form of words(fuzzy search):-''' In Apache Solr Text fields are typically indexed by breaking the text into words and applying various transformations such as lower casing removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed.So when you are searching for &quot;Fly&quot; it also looks for &quot;Flies&quot; or &quot;Flew&quot;<br /> <br /> <br /> * '''Solr with intra-word delimiters:-''' The filter 'WordDelimiterFilter' can be used in the analyzer for the field being queried to match words with intra-word delimiters such as dashes or case changes. By default apache solr provides lots of filters for text field type to configure these things. Properly configured schema.xml is the key point to handle this.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=GSOC/2013&diff=40730 2013-06-06T18:30:33Z

<p>Shashitechno: </p> <hr /> <div>[[Image:banner-gsoc2013.png|right|400px]]<br /> <br /> [[GSOC|GSoC]] &gt; GSoC 2013<br /> <br /> An overview of the [http://www.google-melange.com/gsoc/program/home/google/gsoc2013 Google Summer of Code 2013] projects for Moodle..<br /> <br /> ==Self-assessment activity using the question bank==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1580475&amp;course=5 Jayesh Anandani]<br /> :'''Mentors''': [https://moodle.org/user/profile.php?id=8026&amp;course=5 Jean-Michel Vedrine] &amp; [https://moodle.org/user/view.php?id=93821&amp;course=5 Tim Hunt]<br /> :'''Specification''': [[Self-assessment activity using the question bank]]<br /> See also:<br /> * CONTRIB-4334<br /> * Developer blog link coming<br /> <br /> ==SCORM player rewrite==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1277285&amp;course=5 Mayank Gupta]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> :'''Specification''': [[SCORM player rewrite]]<br /> See also:<br /> * MDL-39910<br /> * Developer blog link coming<br /> <br /> ==Global search==<br /> <br /> :'''Student''': [https://moodle.org/user/view.php?id=1565904&amp;course=5 Prateek Sachan]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=353008&amp;course=5 Tomasz Muras] &amp; [https://moodle.org/user/view.php?id=1146834&amp;course=5 Aparup Banerjee] <br /> :'''Specification''': [[Global search]]<br /> See also:<br /> * MDL-31989<br /> * Developer blog link coming<br /> <br /> ==Course search==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> :'''Mentor''': [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/view.php?id=381842&amp;course=5 Michael de Raadt]<br /> :'''Specification''': [[Course search]]<br /> See also:<br /> * CONTRIB-4335<br /> * [http://shashitechno.wordpress.com/ Developer blog]<br /> <br /> ==Pronunciation evaluation question type==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1594272 Troy Lee]<br /> :'''Mentors''': [https://moodle.org/user/view.php?id=1081865&amp;course=5 James Salsman] &amp; [https://moodle.org/user/view.php?id=93821&amp;course=5 Tim Hunt]<br /> :'''Specification''': [[Pronunciation evaluation question type]]<br /> See also:<br /> * CONTRIB-4336<br /> * [http://pronunciationeval.blogspot.sg/ Developer blog]<br /> <br /> ==BioAuth: A Moodle plugin for determining Quiz authorship==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1595676 Vinnie Monaco]<br /> :'''Mentor''': [https://moodle.org/user/view.php?id=21591&amp;course=5 Dan Marsden]<br /> :'''Specification''': [[BioAuth Plugin]]<br /> See also:<br /> * CONTRIB-4337<br /> * [http://vmonaco.com/blog Developer blog]<br /> <br /> ==Portfolio plugin for Evernote==<br /> <br /> :'''Student''': [https://moodle.org/user/profile.php?id=1459982 Vishal Raheja]<br /> :'''Mentors''': [https://moodle.org/user/profile.php?id=1451616 Frédéric Massart] &amp; [https://moodle.org/user/view.php?id=381842&amp;course=5 Michael de Raadt]<br /> :'''Specification''': [[Portfolio plugin for Evernote]]<br /> See also:<br /> * CONTRIB-4338<br /> * Developer blog link coming<br /> <br /> [[Category:GSOC]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40678 2013-06-05T18:35:47Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335 Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40674 2013-06-05T18:15:48Z

<p>Shashitechno: </p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * [tracker.moodle.org/browse/CONTRIB-4335 CONTRIB-4335]Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40673 2013-06-05T18:14:13Z

<p>Shashitechno: /* Tracker */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * [www.tracker.moodle.org/browse/CONTRIB-4335 CONTRIB-4335]Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40672 2013-06-05T18:13:08Z

<p>Shashitechno: /* Tracker */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * [https://tracker.moodle.org/browse/CONTRIB-4335 CONTRIB-4335]Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40671 2013-06-05T18:12:25Z

<p>Shashitechno: /* See also */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335[https://tracker.moodle.org/browse/CONTRIB-4335]Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * [http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1 GSOC Proposal]<br /> * [https://docs.moodle.org/dev/Projects_for_new_developers#Course_search Original idea]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno https://docs.moodle.org/dev/index.php?title=Course_search&diff=40669 2013-06-05T18:11:06Z

<p>Shashitechno: /* See also */</p> <hr /> <div>{{Work_in_progress}}<br /> {{Infobox Project<br /> |name = Course search<br /> |state = Community bonding period<br /> |tracker = CONTRIB-4335<br /> |discussion = XXX<br /> |assignee = [https://moodle.org/user/profile.php?id=1565325 Shashikant Vaishnav]<br /> }}<br /> &lt;span class=&quot;small-info-right&quot;&gt;<br /> &lt;b&gt;GSOC&lt;/b&gt;<br /> &lt;span class=&quot;text-big new&quot;&gt; '13&lt;/span&gt;<br /> &lt;/span&gt;<br /> <br /> == Introduction ==<br /> <br /> Object of this google summer of code project is to Implement Advanced Course Search that is flexible, case-insensitive, fast and can sort results by relevance. Most importantly it should work consistently on different Database engines and different content languages. The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. The project will be implemented integrating 3rd party search engine API with Moodle course schema.<br /> <br /> == Schedule ==<br /> <br /> '''May 28 - June 17 ( Community Bonding Period)'''<br /> <br /> <br /> ➢ Discuss the further ideas with the mentor .<br /> <br /> ➢ Final list of tasks to be implemented under this project.<br /> <br /> ➢ Read &amp; study the documentations on moodle and its Database Schema.<br /> <br /> ➢ Study the plugin development in moodle.<br /> <br /> ➢ Study the Apache Solr Documentation &amp; its complete working<br /> <br /> ➢ Set up the development and testing environment.<br /> <br /> <br /> <br /> '''June 18- july 10( Interim Period )'''<br /> <br /> <br /> ➢ Start Coding!<br /> <br /> ➢ Prepare moodle course search schema xml configuration<br /> <br /> ➢ Create moodle plugins (admin tool and theme) that allow to substitute core search with alternative solution. At first it will duplicate the existing functionality but it is ready for modifications without changing the core.<br /> <br /> ➢ Create function for manual reindexing of all courses, add button for admin on the plugin page to perform manual reindexing.<br /> <br /> ➢ Implement the search in the plugin just to make sure it works with the simple result output (no filtering, no pagination, no additional information retrieval, etc)<br /> <br /> ➢ Document the work.<br /> <br /> <br /> '''July 11- August 02( Interim Period )'''<br /> <br /> <br /> ➢ Display courses properly, filtered, with all additional information<br /> <br /> ➢ Add caching of search results so everything is not retrieved again when user requests the second page<br /> <br /> ➢ Prepare search plugin for distrubution: create install/upgrade/uninstall scripts, write Readme file how to install and configure Solr.<br /> <br /> <br /> <br /> Mid Term Evaluation<br /> <br /> Submit code of completed tasks along with documentation.<br /> <br /> <br /> '''August 3- August 25 (Interim period)'''<br /> <br /> <br /> ➢ Listen to events and mark courses for reindexing<br /> <br /> ➢ Implement cron function that does reindexing<br /> <br /> ➢ Add configurable settings to the plugin (when/whether to run automatic re indexing)<br /> <br /> ➢ Document the work.<br /> <br /> ➢ Debug the code and reduce code complexity.<br /> <br /> <br /> '''August 26- Sep 16 (Interim period)'''<br /> <br /> <br /> ➢ Filtering results by dates, tags.<br /> <br /> ➢ Spell correction capability.<br /> <br /> ➢ Fuzzy search support.<br /> <br /> ➢ Implement extra functionality (if any other feature required and time permits).<br /> <br /> <br /> '''Sep 17- Sep 23(Pencils down)'''<br /> <br /> <br /> ➢ Testing, documentation &amp; debugging.<br /> <br /> ➢ Final Release<br /> <br /> == Requirements ==<br /> <br /> The deliverable includes a functional advance search plugin for Moodle that can be installed and configured to substitute basic core search. Usually require moodle version 2.6.<br /> <br /> == Features ==<br /> <br /> <br /> * Non latin support with search queries.<br /> <br /> <br /> * Indexing to make course search fast and efficient <br /> <br /> * Sorting by relevance <br /> <br /> * spelling correction (Did you mean?) feature.<br /> * fuzzy search (alternate form of words).<br /> <br /> * Work consistently on different database engines and different content language.<br /> <br /> == Design ==<br /> <br /> '''Relevancy Search wireframe:-'''<br /> <br /> The following wireframe shows how the results can be filtered with different aspects.<br /> <br /> [[File:Wireframe.png]]<br /> <br /> == Credits ==<br /> <br /> * '''Mentors:''' [https://moodle.org/user/profile.php?id=1334243&amp;course=5 Marina Glancy] &amp; [https://moodle.org/user/profile.php?id=381842 Michael de Raadt]<br /> <br /> == Tracker ==<br /> <br /> * CONTRIB-4335[https://tracker.moodle.org/browse/CONTRIB-4335]Course Search Plugin<br /> <br /> == See also ==<br /> * [[GSOC/2013|Moodle GSoC projects for 2013]]<br /> * [[GSOC|Moodle GSoC Overview page]]<br /> * GSOC Proposal[[http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shashitechno/1]]<br /> * Original idea[[https://docs.moodle.org/dev/Projects_for_new_developers#Course_search]]<br /> <br /> [[Category:GSOC]]<br /> [[Category:Project]]</div>

Shashitechno