Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Global search (GSoC2013): Difference between revisions

From MoodleDocs
Line 152: Line 152:
*All Lesson names.
*All Lesson names.
*Lesson Media files. (To be discussed with my mentors).
*Lesson Media files. (To be discussed with my mentors).
*Lesson pages within each lesson. (Currently, the lesson-pages may not be indexed/searchable due to this bug: [https://tracker.moodle.org/browse/MDL-40420 MDL-40420]. Once, this gets integrated, it will work appropriately.)
*Lesson pages within each lesson. (Currently, the lesson-pages may not be indexed/searchable due to this bug: [https://tracker.moodle.org/browse/MDL-40420/ MDL-40420]. Once, this gets integrated, it will work appropriately.)
**Content Page: Page's title and content.
**Content Page: Page's title and content.
**Cluster: Cluster's title and content.
**Cluster: Cluster's title and content.

Revision as of 10:51, 26 July 2013

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Global search
Project state Coding period
Tracker issue MDL-31989
Discussion Writing Moodle's Global Search
Assignee Prateek Sachan

GSOC '13

Introduction

Global Search will have the feature of searching keywords within the entire Moodle site across modules keeping the security intact.

  • It will display results based on relevance weightage.
  • Security will be preserved throughout the search.
  • Search Modules will enable chosen search engine integration with ease. Admins will have the option for selecting the modules that could be made "searchable"
  • It will include keywords from other files types (like PDFs, PPTs, HTML content and others).
  • Following are the features that I'm considering in implementing in the first version of Global Search:
  1. Groupings of boolean operators:AND, OR, NOT. Eg.: ("query1" AND "query2") OR ("query3" NOT "query4")
  2. Searching for phrases. Results with matched phrase will have higher priority and hence will be shown higher in the results.
  3. Wildcard (*) (?) feature.
  4. Stemming. Eg.: bag will return results both from bag and bags
  5. Proximity Searches:
  • "mood"~2 returns "moodle". (2 alphabets away from the searched term).
  • "moodle australia"~3 returns results containing "moodle hq at perth australia" (the queried terms were within 3 words of each other)

Requirements

  • Moodle 2.5 and above.
  • PHP Solr extension.

Downloading PHP Solr extension

Downloading extension for Apache Solr 4.x

Apache Solr 4.x requires installation of php solr extension version 1.0.3-alpha. You will be ready to install the PECL extension for Solr by cloning the following repository for Solr 4.x. (Please Note: Currently, the official php-pecl-solr is not compatible with Solr 4.x. The following repository provides a small fix to make it compatible with Solr 4.x and will go to the official release).

git clone https://github.com/lukaszkujawa/php-pecl-solr.git

Downloading extension for Apache Solr 3.x

Apache Solr 3.x requires installation of php solr extension version <=1.0.2. You can download the official latest versions from http://pecl.php.net/package/solr. Extract the contents into a directory.

Installing the downloaded PHP Solr extension

For using Global Search, users will have to install the PHP Solr PECL extension on server. Users will have the option of configuring Solr version in Global Search.

  • Quick install using precompiled packages for UNIX* systems (Debian/Ubuntu) and MS Windows(tm).

Following is the procedure for installing the downloaded extension in UNIX:

There are two dependencies of the extension:

  • CURL extension (libcurl 7.15.0 or later is required)
  • LIBXML extension (libxml2 2.6.26 or later is required)

If the system does not have required versions of libcurl or libxml libraries, follow the steps given below. You will have to download the libraries and compile them from source into a separate install prefix.

  1. For libcurl:
    wget http://curl.haxx.se/download/curl-7.19.6.tar.gz

tar -zxvf curl-7.19.6.tar.gz cd curl-7.19.6 sudo ./configure --prefix=/root/custom/software sudo make sudo make install

  1. For libxml:
    wget ftp://xmlsoft.org/libxml2/libxml2-2.7.6.tar.gz

tar -zxvf libxml2-2.7.6.tar.gz cd libxml2-2.7.6 sudo ./configure --prefix=/root/custom/software sudo make sudo make install After installing the above dependencies, you will need to restart your apache server by executing sudo service apache2 restart

Next,

  • cd /your-downloaded-or-cloned-directory/
  • phpize
    • This a shell script used to prepare the build environment for a php extension to be compiled. If you don't have phpize, you can install it by executing sudo apt-get install php5-dev

If the libxml and libcurl libraries were compiled from source, then you will have to pass the libcurl prefix to the configure script for CURL and LIBXML respectively as shown below:

  • sudo ./configure --enable-solr --with-curl=/root/custom/software --with-libxml-dir=/root/custom/software
  • If you already have the latest versions of the libraries then executing sudo ./configure is sufficient.
  • sudo make
  • sudo make install

The above procedure will compile and install it in the extension_dir directory in the php.ini file. To enable, the installed extension, you could follow any of the following two steps:

1. Navigate to the directory /etc/php5/conf.d and create a new solr.ini file with the following line:

extension=solr.so

OR

2. Open your php.ini file and include the following line:

extension=solr.so

You may follow any of the above two steps. You will need to restart your apache server after that by executing sudo service apache2 restart

You can now view the solr extension details by clicking PHP info from Site administration > Server in browser or php -m in Terminal (Ctrl+Alt+T)

Download and Installation - OSX using macports

This method provides an easy install of php solr extension without any downloads.(php solr extension version: <=1.0.2)

- sudo port install apache-solr4
- sudo port install php54-solr

you can choose your relevant available versions @ http://www.macports.org/ports.php?by=name&substr=solr

Setting up Global Search in Moodle after installing PHP Solr extension

After installing the php-pecl-solr extension, users will have to download the required Apache Solr release (version 4.x for solr-php extension 1.0.3-alpha or 3.x for solr-php extension version <=1.0.2), unzip it and keep it in an external directory of Moodle.

Users will have to replace solconfig.xml and schema.xml inside the downloaded directory /example/solr/conf with the ones that Global Search will provide in /search/solr/conf/ directory. Once, the files have been copied and replaced, users will have to start the java jetty server start.jar located in /example/ directory by executing java -jar start.jar.

Searchable content in Global Search

This section illustrates the content that will be indexed to be made searchable. Necessary security will be implemented at appropriate places.

Book Resource

This section outlines the content that will be indexed and made searchable from within the Book Resource.

  • Name of Book.
  • Introductory content of the book.
  • Title of all chapters within the book.
  • Content of all chapters within the book.

The search results will give a link to the chapter.

Forum Module

This section outlines the content that will be indexed and made searchable from within the Forum Module.

  • All forum posts.
  • User's name who have posted on the forum.
  • Attached files in the posts. (PDFs, PPTX, etc.) (Necessary security will be checked. Users will not be able to see the file's link unless they have the necessary capability to do so.)

The search results will give a link to a forum's post.

Glossary Module

This section outlines the content that will be indexed and made searchable from within the Glossary Module.

  • Glossary
    • Names.
    • Introductory content.
  • Glossary entries within each glossary
    • Title.
    • Content.
    • User.
    • Attached files. (PDFs, PPTX, etc.)

The search results will give a link to a particular glossary entry.

Label Resource

This section outlines the content that will be indexed and made searchable from within the Label Resource.

  • All Label names.
  • Introductory content.

The search results will give a Label's course link.

Lesson Module

This section outlines the content that will be indexed and made searchable from within the Lesson Module.

  • All Lesson names.
  • Lesson Media files. (To be discussed with my mentors).
  • Lesson pages within each lesson. (Currently, the lesson-pages may not be indexed/searchable due to this bug: [https://tracker.moodle.org/browse/MDL-40420 MDL-40420]. Once, this gets integrated, it will work appropriately.)
    • Content Page: Page's title and content.
    • Cluster: Cluster's title and content.
    • Question: Question's title and Content.

The search results will give a link to the lesson page.

Page Resource

This section outlines the content that will be indexed and made searchable from within the Page Resource.

  • Name of Page.
  • Introductory content of Page.
  • Content of Page.

The search results will give a link to the page.

File Resource

This section outlines the content that will be indexed and made searchable from within the File Resource.

  • Name of File resource.
  • Introductory content of File resource.
  • Uploaded rich document (set as main file) in the File resource.

The search results will give a link to the File resource page.

Url Resource

This section outlines the content that will be indexed and made searchable from within the Url Resource.

  • Name of Url.
  • Introductory content of Url.
  • External link of Url resource.

The search results will give a link to the Url Resource.

Wiki Module

This section outlines the content that will be indexed and made searchable from within the Wiki Module.

  • Name of Wiki.
  • Introductory content of Wiki.
  • Comments. (To be discussed with my mentors)
  • Files.(To be discussed with my mentors)
  • All Wiki Sub-Pages.
    • Title of Wiki Sub-Pages.
    • Content of Wiki Sub-Pages.

The search results will give a link to the wiki sub-page.

Implementation and Milestones

  • Writing cron jobs:(17th June - 21st June)
    • Addition of records.
    • Deletion of records.[shouldn't be focused upon just now]
    • Update of records.
  • Design the solr schema and solconfig files. (22nd June - 26th June)
    • These files will be embedded in the in a separate directory under the Global Search directory. Users will have to copy these two files to the Apache Solr example directory that they will download which will run the Solr jetty server. See Installation for more.
    • schema.xml contains all the properties about the documents fields which are being indexed. There may be different fields pertaining to different modules.
    • solrconfig.xml contains the configurational parameters for solr.
  • Writing the core search API for all searchable modules. (27th June - 3rd July)
    • _SEARCH_ITERATOR($from=0)
    • _SEARCH_DOCUMENTS($id)
  • Adding proper security to the search API.(4th July - 8th July)
    • _SEARCH_ACCESS($id)
      • Use of Access API
        • ACCESS_DENIED
        • ACCESS_GRANTED
      • ACCESS_DELETED: Situations where the records may have been deleted-hence not viewable.
  • Reviewing all the above once. (9th July - 10th July)
  • Integrating Apache Tika to handle indexing from external files (PDFs, PPTX etc.) (11th July - 13th July)
  • Writing the search functions for querying. (Just a basic search UI to be used at this moment). (14th July - 17th July)
    • Input for query.
    • Input for filter fields for filtering the search results.
    • Input for AND/OR.
    • Phrase searches.
    • Stemming.
    • Support for wildcards.
  • Admin page for search configuration options. (18th July - 21st July) (The UI page here will be the default type as being currently used. For example, Site Administration>Advance features)
    • Deletion of index.
      • Deletion of entire index in one go.
      • Deletion of index by specific modules only.(For example, only the index of records belonging to 'forum' module is to be deleted).
    • configurational options for cron
      • Time of cron run.
  • Implementing and releasing the first prototype version: 1.0 for developers' feedback. (22nd July - 28th July)
    • See the Prototype section.
  • Preparing for mid-term evaluation. (29th July - 1st August)
  • MID_TERM EVALUATION (2nd August)
  • Re-designing the search page. (2nd August - 7th August) (Taking ideas from community+discussion in forum)
  • Improving the prototype after feedback from developers.(8th August - 15th August)
    • Bug-fixing.
    • Fixing security leaks.
    • Improving relevance & speed of search results.
  • Running Test cases and performance testing. (16th August - 21st August) (Performance testing will be good at this point as the code would have been optimized to some level as instructed by the developers above)
  • Debugging. (22nd August - 29th August)
  • Finalizing the Global Search documentation. (30th August - 6th September)
    • Discussing it with my mentors whether everything has been properly covered or not.
  • Buffer Period. (7th September - 8th September)
    • Making sure everything above has been implemented correctly and efficiently.
  • Submitting my code to Moodle and Google. (9th September - 15th September)
  • Suggested Pencils Down Period. (16th September)
  • Performing edits to the documentation after feedback from the Moodle community. (17th September - 22nd September)
  • Firm Pencils Down and Final Evaluation. (23rd September)

Prototype: version 1.0

This covers the features and procedure to install Global Search plugin. It would be very good if developers may come forward to test it and give their feedback which would be very crucial for improving it. Developers may check out the cases in the Testing section. Developers may add their own preferred test cases and results (pass/fail) in that section which would be helpful for the other developers or you may comment on Global Search discussion on Developer Forum.

  • Here are some things that I've summarized that may be focused upon:
    • Finding out any security leaks that I may have missed out.
    • Relevance/speed of displaying search results.
    • Taking ideas to optimize the source code wherever possible.
    • Getting feedback.
    • Addition/deletion of any search features that you may feel would be useful.
  • Features included in this prototype:
    • Indexing/Searching of content in all the above mentioned modules/resources.
    • Support for indexing/searching rich documents.
    • Admin configuration options.
    • Support for advanced search queries as stated above.
  • Steps to install and test Global Search plugin:
    • Make sure you've installed your preferred PHP Solr extension (see sections above).
    • You can simply clone my Github branch and checkout gs2 branch.

git clone https://github.com/prateeksachan/moodle.git git checkout gs2

    • Go to Setting up Global Search in Moodle after installing PHP Solr extension section.
    • Open ../moodle/admin in your browser, and update the plugin.
    • Indexing is through cron. You will have to run the cron script to index content.

Search workflow

Simple version

  1. Moodle sends a query to standard solr handler.
  2. solr returns 1000 results. Results include all fields required to render search result row. Results are ordered by relevancy.
  3. Moodle checks each result in order to establish if it should be visible for current user or not
  4. Once 100 visible results are found, Moodle stops checking the rest and displays 100 results

The best case scenario is that Moodle needs to check only 100 returned results (first 100 are accessible for the current user). The worst case scenario is that out of 1000 returned documents, none of them is available for the current user. In this case Moodle performs 1000 checks and displays "no results found" message.

Advanced version

More advanced version will use a logic on solr side to pre-filter some of the results.

Testing

A list of test cases.

New content

  • index
  • add new forum post
  • re-index
  • make sure new content is searchable

Updated content

  • index
  • edit existing forum post
  • re-index
  • make sure new content is searchable
  • make sure old content is not searchable

Deleted content

  • index
  • delete existing forum post
  • re-index
  • make sure deleted content is not searchable

Backup restored

  • index
  • restore whole course from a backup
  • re-index
  • make sure restored content is searchable

Access Tests

Common Access

  • Add courses.
  • Create activity modules/resources that are supported by Global Search (See section Searchable content in Global Search).
  • Insert content or attach files (wherever possible)
  • Index
  • Make sure the results are visible only to those who have access to the respective course's activity/resource.
  • Set Common Module Settings of activities/resources as Hidden.
  • Make sure those activities/resources do not appear in search results.

Forum Activity Module Access

  • Create a course.
  • Create groups in it.
  • Create a Forum Activity.
  • Post in the forum discussions from members of different groups. (+you may also attach files)
  • Index.
  • Make sure members see posts only from those groups which they are a member of.

Lesson Activity Module Access

  • Create a course.
  • Create a Lesson Activity.

1. Type 1

  • Assign Availability
    • Available From only
    • Deadline only
    • Time limit
    • Password Protection
    • Combinations of the above
  • Index.
  • Make sure there are no access leaks when searching as a student.
  • Make sure the teacher, manager are able to see the searches.

2. Type 2

  • Assign Prerequisite Lesson
    • Set Dependent On Prerequisite feature.
    • Set Time spent Prerequisite feature.
    • Toggle completed Prerequisite feature.
    • Set Grade Prerequisite feature.
    • Combinations of the above.
  • Index
  • Make sure there are no access leaks when searching as a student.
  • Make sure the teacher, manager are able to see the searches.

Wiki Activity Module Access

  • Create a course.
  • Create Groups.
  • Create a Wiki Activity.
  • Set Group Mode On

1. Type 1

  • Group Mode: Separate Groups
  • Add Wikipages. (+you may also attach files)
  • Index
  • Make sure there are no access leaks: A group shouldn't see results from other groups.

2. Type 2

  • Group Mode: Visible Groups
  • Add Wikipages.(+you may also attach files)
  • Index
  • Make sure there are no access leaks: A group shouldn't see results from other groups.

Results

Credits

See also