Global search: Difference between revisions

Latest revision as of 14:47, 9 June 2022

What is Global search?

Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.
The feature needs to be enabled by the administrator in Advanced features and then, from the Manage Global search page, the search engine must be selected.

When Global search is enabled, search box is then available next to the user menu.

Searching the site (when global search is enabled)

A Global search block may also be added to course pages.

What can I search for?

You can search for your courses, activities information and some activities contents like forum posts, book chapters, glossary entries or collaborative wikis pages.

HTML block content may be searched and, if the administrator sets the setting "Searchable courses" to Search within all courses the user can access then courses which are visible to users but which they are not enrolled in may also be searched. (These courses are courses with guest access or if the user has the capability to view all courses.)

How does it work?

Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the Global search block if enabled.
On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:

Filtering the search

You will then see results displayed from all areas of Moodle you have access to:

Search results

Solr-specific features

These features are only available if you use Solr as your search engine.

Search query features

You can improve your search using any of the following search query features:

Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title
Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords
Wildcard characters ('*' or '?' ) to represent characters in the search query
Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)
Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.

File contents indexing

Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts.

Extra filters

New in 3.5 : New fields now facilitate filtering by user ID and searching by group.

How is it set up?

Note: Global search needs a search engine. It's recommended you set everything up first, index the site contents and only then enable Global search.

Moodle core includes two search engines: Simple Global search and Solr. See the Developer docs on Search engine plugins if you wish to write your own.

Setting up Simple search

Simple Global search provides global search features without the need to install an external search engine.

Simple global search

Setting up Solr

The latest Solr 5 available version is the recommended one.
Moodle supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released.

General Setup

Set the feature up in Site administration > Plugins > Search > Manage global search by selecting Solr as the search engine and ticking search area checkboxes as required
In Site administration > Plugins > Search > Solr, set Host name to localhost, Port to 8983 and Index name to 'moodle' (the name of the index in Solr)
If you are using Solr with SSL encryption and basic authentication (see below if you want to use client certificates instead), you will need to configure this.This is important if solr is not installed in the web server server.
1. Go to Site administration > Plugins > Search > Solr
2. Set Secure mode to Yes
3. Set port number if required (sometimes SSL installations use port 443).
4. Type in the username and password.
If you are using Solr with SSL encryption via client certificates, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.
1. You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.
2. Go to Site administration > Plugins > Search > Solr
3. Set Secure mode to Yes
4. SSL certificate to /path/to/certs/solr-ssl.cacert.pem
5. SSL key to /path/to/certs/solr-ssl.key.pem
6. SSL key Password to The password used to lock the SSL Key
7. SSL CA certificates name to /path/to/certs/solr-ssl.cacert.pem
You now need to populate the created Solr index with your site's data. You can do it via the web interface by going to Site administration > Reports > Global search info or from the CLI by running the search/cli/indexer.php script. The CLI script is the recommended option for big sites.
# sudo -u www-run php search/cli/indexer.php --force
Enable Global search in Site administration > Advanced features

File Indexing

Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the Tika engine which comes bundled with Solr. To enable this feature:

In Site administration > Plugins > Search > Solr enable the checkbox Enable file indexing
Set Maximum file size to index (kB) to some value - the default is 2097152 (2GB)
- Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.

New in 34: File indexing now works retroactively -i.e., if you set up Global search without file indexing enabled and a later date you enable file indexing, files associated with existing objects will be included.

How to install Solr

You need PHP Solr extension installed. You can download the official latest versions from PECL: Package: solr. The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.

Basic installation steps (using Apache web server):

Linux (Debian/Ubuntu)

Using APT:

   sudo apt install php-solr
   sudo service apache2 restart

If you can't install the SOLR PHP module via APT, try the following:

With PHP5.x

   sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev
   sudo apt-get install php5-dev
   sudo apt-get install php-pear
   sudo pecl install solr
   sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"
   sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"
   sudo service apache2 restart

With PHP 7

   sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev
   sudo apt-get install php7.0-dev
   sudo apt-get install php-pear
   sudo pecl install solr
   sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/apache2/conf.d/solr.ini"
   sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/cli/conf.d/solr.ini"
   sudo service apache2 restart

Linux (Redhat/Centos 6 & 7)

Using built in php5 packages

 yum install php-pecl-solr2
 service httpd restart

Using 3rd party php7 packages (webtactic)

 yum install libxml2-devel pcre-devel libcurl-devel php70w-devel php70w-pear
 pecl install solr
 echo "extension=solr.so" > /etc/php.d/solr.ini
 service httpd restart

Linux (Amazon Linux 1)

AWS AMIs do not include the solr extension, but you are still able to compile the extension from source using pecl.

NOTE: this method will only build the solr extension for a single version of PHP (php7.3 in this example).

   sudo yum -y groupinstall "Development Tools"
   sudo yum -y install php-pear php73-devel curl-devel libxml2-devel
   sudo pecl channel-update pecl.php.net
   
   # Locate /usr/share/pear/Archive/Tar.php
   # Line 639
   # Change
   # $v_att_list = & func_get_args();
   # to
   # $v_att_list = func_get_args();
   
   # Choose desired php version as default
   sudo alternatives --config php
   
   # Set specific temp directory
   sudo mkdir -pv /build/tmp
   sudo pear config-set temp_dir /build/tmp
   sudo touch /etc/php-7.3.d/solr.ini
   sudo pear config-set php_ini /etc/php-7.3.d/solr.ini
   
   sudo pecl install solr-stable

OSX using macports

   sudo port install apache-solr4
   sudo port install php54-solr

OSX using homebrew

   brew install homebrew/php/php56-solr

Windows

Check your PHP version (see Site admin > Server > PHP Info) for PHP version, compiler (e.g. VC15), architecture (e.g. x64), and thread safety.
Get the matching PHP extension DLL from PECL (or this page if not available on PECL).
Add the DLL to you php\ext folder.
Add the directive "extension=php_solr.dll" to your php.ini file.
Restart your web server.
Install the Java Runtime Environment
Download Solr from the Solr Downloads page, e.g. solr-5.5.5.zip.
Extract to an accessible place, e.g.: c:\solr-5.5.5
From command line (as admin), in C:\solr-5.5.5, run...
- bin\solr start
- bin\solr status
- bin\solr create -c moodle
- bin\solr stop -all
(As admin) Edit C:\solr-5.5.5\server\solr\moodle\conf\solrconfig.xml and change line...

<maxBooleanClauses>1024</maxBooleanClauses>

...to...

<maxBooleanClauses>524288</maxBooleanClauses>

Create Scheduled task to start Solr
- General
  - Run whether user is logged on or not
  - Run at highest privileges
  - Configure for Windows Server 2016
- Triggers
  - Run daily (start any time)
  - Repeat every hour, indefinitely
- Actions
  - Start program
  - Script: C:\solr-5.5.5\bin\solr.cmd
  - Add arguments: start -m 2g
- Conditions
  - Uncheck: Start the task only when the computer is on AC power
- Settings
  - Uncheck: Stop the task if it runs longer than: ...
  - If the task is already running, then the following rule applies: Do not start a new instance
Run the scheduled task manually (for now)
At the command line, in c:\solr-5.5.5, run bin\solr status to check it is running
Configure the Solr server in Moodle as below.
Visit Site admin > Plugins > Search > Manage global search. It may prompt you to set up the search schema.
Run the scheduled task to build the search index from the command line. It may initially need to run a few times as it will time out after 10min and you may have a lot of files.

php admin\tool\task\cli\schedule_task.php --execute=\core\task\search_index_task

Keep an eye on the amount of memory used with solr status and the Task Manager (see the Java process).

The Solr server

Note that for medium/large sites you may need to increase maxBooleanClauses setting. In [https://tracker.moodle.org/browse/MDL-54992 MDL-54992] we are working on an alternative way to query the server.

Please note that there is a security issue that affects some Solr versions. The patched versions are 5.5.5, 6.6.2, 7.1, 7.2 and all new versions are patched. If your Solr server version is lower but you can't upgrade to one of the patched versions it is recommended to disable the XML Query Parser.

The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.5.5 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named moodle to add moodle data to it. If wget gives an error, check http://www-us.apache.org/dist/lucene/solr and update SOLRVERSION

   #!/bin/bash
   set -e
   SOLRVERSION=5.5.5
   SOLRNAME=solr-$SOLRVERSION
   SOLRTAR=$SOLRNAME'.tgz'
   INDEXNAME=moodle
   if [ -d $SOLRNAME ]; then
       echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."
       exit 1
   fi
   if [ ! -f $SOLRTAR ]; then
       wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR
   fi
   tar -xvzf $SOLRTAR
   cd $SOLRNAME
   bin/solr start
   bin/solr create -c $INDEXNAME
   # After setting it up and creating the index use:
   # - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server
   # - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.

Solr 5/6 schema setup

Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.

For very large or busy sites, it is recommended that you manually remove the _text_ field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.

Solr 4 schema setup

You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:

Extracted from search/classes/document.php

Field name	Field type	Stored	Indexed	Query field
id	org.apache.solr.schema.StrField	true	false	false
itemid	org.apache.solr.schema.TrieIntField	true	true	false
title	org.apache.solr.schema.TextField	true	true	true
content	org.apache.solr.schema.TextField	true	true	true
contextid	org.apache.solr.schema.TrieIntField	true	true	false
areaid	org.apache.solr.schema.StrField	true	true	false
type	org.apache.solr.schema.TrieIntField	true	true	false
courseid	org.apache.solr.schema.TrieIntField	true	true	false
owneruserid	org.apache.solr.schema.TrieIntField	true	true	false
modified	org.apache.solr.schema.TrieDateField	true	true	false
userid	org.apache.solr.schema.TrieIntField	true	true	false
description1	org.apache.solr.schema.TextField	true	true	true
description2	org.apache.solr.schema.TextField	true	true	true
solr_filegroupingid	org.apache.solr.schema.StrField	true	true	false
solr_fileid	org.apache.solr.schema.StrField	true	true	false
solr_filecontenthash	org.apache.solr.schema.StrField	true	true	false
solr_fileindexstatus	org.apache.solr.schema.TrieIntField	true	true	false
solr_filecontent	org.apache.solr.schema.TextField	false	true	true

Memory and File indexing considerations

For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, e.g. aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the -m option:

solr start -m 12g

See the documents for your version of Solr on how to increases memory when Solr is started automatically.

Too many boolean clauses error

Because of the way Moodle handles permissions for searches, if you have non-admin users with access to a large number of activities (>1000), they may an error similar to the following:

   The query you provided could not be parsed by the search engine: org.apache.solr.search.SyntaxError: Cannot parse 
   ...
   too many boolean clauses

To correct this, you need to increase the maxBooleanClauses setting in your Solr core. The setting is located in corename/conf/solrconfig.xml.

This important note is included in the config file:

   ** WARNING **
   
   This option actually modifies a global Lucene property that
   will affect all SolrCores.  If multiple solrconfig.xml files
   disagree on this property, the value at any given moment will
   be based on the last SolrCore to be initialized.

This means that for consistent behavior you should update this value for all cores in the Solr server.

Command for installing Solr Server on Centos 7

First of all install java (openjdk)

yum install java-1.8.0-openjdk

Add user for solr

adduser solr

Download solr server, find latest here : http://lucene.apache.org/solr/mirrors-solr-latest-redir.html, from the list there download latest solr-[version].tgz

Here is the list of needed commands (note that latest solr is suggested, change command to appropriate version)

cd /opt
wget http://www-eu.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz
tar zxvf solr-6.6.0.tgz
cp solr-6.6.0/bin/install_solr_service.sh .
rm -rf solr-6.6.0
./install_solr_service.sh solr-6.6.0.tgz
chkconfig solr on

Create the index

su - solr -c "/opt/solr/bin/solr create_core -c moodle"

Then go to your Moodle and set index name as created (moodle)

Documentation