https://docs.moodle.org/310/en/api.php?action=feedcontributions&feedformat=atom&user=DmonllaoMoodleDocs - User contributions [en]2026-04-20T04:50:55ZUser contributionsMediaWiki 1.43.5https://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=136442Analytics settings2020-01-15T03:51:03Z<p>Dmonllao: /* AWS serverless service */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
<br />
{{New features}}Analytics may be disabled from Site administration / Advanced features<br />
<br />
They may then be configured from Site administration / Analytics. <br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
==== PHP predictions processor ====<br />
<br />
The PHP processor is the default. There are no other system requirements to use this processor.<br />
<br />
==== Python predictions processor ====<br />
<br />
The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package. The package can be installed in the web server (in all the codes if in a clustered environment) or in a separate server.<br />
<br />
===== Installed in the web server ===== <br />
<br />
The latest version of the package for '''Moodle 3.8''' is compatible with '''Python 3.4, 3.5, 3.6 and 3.7'''. Note that the package should be available for both the Command Line Interface (CLI) user and the user who runs the web server (e.g. www-data).<br />
<br />
# If necessary, install Python 3 (and pip for Python 3)<br />
# Ensure that you use Python 3 to install the moodlemlbackend package:<br />
<br />
sudo -H python3 -m pip install "moodlemlbackend==2.3.*"<br />
# You must also enter the path to the Python 3 executable in Site administration -> Server -> System paths:<br />
[[File:path_to_python_3.png|frame|center|Enter system path for Python 3]]<br />
<br />
===== Installed in a separate server =====<br />
<br />
To install the python package in a separate server instead of installing it on the web server/s have some advantages:<br />
* Keeps the python ML backend as an external service<br />
* To keep a separated control of the resources the web server/s dedicate to serving Moodle and the resources dedicated to the python ML backend<br />
* You can reuse the same ML server for multiple Moodle sites. Easier to setup and maintain than to install/upgrade the python package in all nodes in the cluster<br />
* You can install the package as a new docker container in your dockerized environment<br />
* You can serve the ML backend from AWS through the API gateway and AWS lambda, storing the trained model files in S3<br />
<br />
On the other hand, it is expected that there can be some added latency in connecting to the python ML backend server.<br />
<br />
The python backend is exposed as a Flask application. The Flask application is part of the official 'moodlemlbackend' Python package and its FLASK_APP script is 'webapp', in the root of the package. You are free to use the setup that better suits your existing infrastructure.<br />
<br />
====== New server in your infrastructure ======<br />
<br />
The python ML backend is exposed as a Flask application, which uses a WSGI server (https://wsgi.readthedocs.io/en/latest/what.html) to be exposed to the www. The official documentation on how to deploy a Flask app can be found in https://flask.palletsprojects.com/en/1.0.x/tutorial/deploy/.<br />
<br />
* Use MOODLE_MLBACKEND_PYTHON_USERS environment var to set a list of users and password (comma-separated). The value is 'default:sshhhh' (user: default, password: sshhhh).<br />
* Set MOODLE_MLBACKEND_PYTHON_DIR to the path you want to use to store the data generated by the package<br />
<br />
====== Docker ======<br />
<br />
https://hub.docker.com/r/moodlehq/moodle-mlbackend-python is the official moodle-mlbackend-python docker image. We use it internally at Moodle HQ for internal testing and you can use it as well. You may want more control over the image, if that is the case https://github.com/moodlehq/moodle-docker-mlbackend-python/blob/master/Dockerfile can serve as an example of what is needed to get the python moodlemlbackend package working.<br />
<br />
* Use MOODLE_MLBACKEND_PYTHON_USERS environment var to set a list of users and password (comma-separated). The value is 'default:sshhhh' (user: default, password: sshhhh).<br />
<br />
====== AWS serverless service ======<br />
<br />
You can serve the Flask application as a serverless application using the AWS API gateway and AWS lambda. The easiest way to do it is using Zappa https://github.com/Miserlou/Zappa to deploy the Flask application contained in the python ML package.<br />
<br />
* Use MOODLE_MLBACKEND_PYTHON_USERS environment var to set a list of users and password (comma-separated). The value is 'default:sshhhh' (user: default, password: sshhhh).<br />
* Set MOODLE_MLBACKEND_PYTHON_DIR to the path you want to use to store the data generated by the package<br />
* Set these environment variables below to setup the S3 access:<br />
** MOODLE_MLBACKEND_PYTHON_S3_BUCKET_NAME to the bucket name<br />
**AWS_ACCESS_KEY_ID as usual<br />
** AWS_SECRET_ACCESS_KEY as usual<br />
<br />
Once this has been done, you can select the Python prediction processor as the default or for an individual model:<br />
[[File:python_backend.png|frame|center|Predictions processor default]]<br />
<br />
[[File:python_backend_at_model.png|frame|center|Predictions processor selection for an individual model]]<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
Several analysis intervals are available for models in the system. In this setting, the analysis intervals that will be used [[Using_analytics#Evaluating_models|to evaluate models]] are defined, e.g. so the best analysis interval identified by the evaluation process can be selected for the model. This setting does not restrict the analysis intervals that can be used for specific models.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Defining roles ==<br />
<br />
Moodle learning analytics makes use of a number of capabilities. These can be added or removed from roles at the site level or within certain contexts to customise who can view insights.<br />
<br />
To receive notifications and view insights, a user must have the [[Capabilities/moodle/analytics:listinsights|analytics:listinsights]] capability within the context used as the "Analysable" for the model. For example, the [[Students at risk of dropping out]] model operates within the context of a course. Insights will be generated for each enrolment within any course matching the criteria of the model (courses with a start date in the past and an end date in the future, with at least one teacher and student), and these insights will be sent to anyone with the listinsights capability in that course. By default, the roles of [[Teacher role|Teacher]], [[Non-editing teacher role|Non-editing teacher]], and [[Manager role|Manager]] have this capability.<br />
<br />
Some models (e.g. the ''No teaching'' model) generate insights at the Site level. To receive insights from these models, the user must have a role assignment at the System level which includes the listinsights capability. By default, this is included in the [[Manager role]] if assigned at the site level.<br />
<br />
'''Note''': Site administrators do '''not''' automatically receive insight notifications, though they can choose to view details of any insight notifications on the system. To enable site administrators to receive notifications of insights, assign an additional role that includes the listinsights capability to the site administrator at the system level (e.g. the [[Manager role]]).<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]<br />
[[de:Analytics-Einstellungen]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=136441Analytics settings2020-01-15T03:49:41Z<p>Dmonllao: /* New server in your infrastructure */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
<br />
{{New features}}Analytics may be disabled from Site administration / Advanced features<br />
<br />
They may then be configured from Site administration / Analytics. <br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
==== PHP predictions processor ====<br />
<br />
The PHP processor is the default. There are no other system requirements to use this processor.<br />
<br />
==== Python predictions processor ====<br />
<br />
The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package. The package can be installed in the web server (in all the codes if in a clustered environment) or in a separate server.<br />
<br />
===== Installed in the web server ===== <br />
<br />
The latest version of the package for '''Moodle 3.8''' is compatible with '''Python 3.4, 3.5, 3.6 and 3.7'''. Note that the package should be available for both the Command Line Interface (CLI) user and the user who runs the web server (e.g. www-data).<br />
<br />
# If necessary, install Python 3 (and pip for Python 3)<br />
# Ensure that you use Python 3 to install the moodlemlbackend package:<br />
<br />
sudo -H python3 -m pip install "moodlemlbackend==2.3.*"<br />
# You must also enter the path to the Python 3 executable in Site administration -> Server -> System paths:<br />
[[File:path_to_python_3.png|frame|center|Enter system path for Python 3]]<br />
<br />
===== Installed in a separate server =====<br />
<br />
To install the python package in a separate server instead of installing it on the web server/s have some advantages:<br />
* Keeps the python ML backend as an external service<br />
* To keep a separated control of the resources the web server/s dedicate to serving Moodle and the resources dedicated to the python ML backend<br />
* You can reuse the same ML server for multiple Moodle sites. Easier to setup and maintain than to install/upgrade the python package in all nodes in the cluster<br />
* You can install the package as a new docker container in your dockerized environment<br />
* You can serve the ML backend from AWS through the API gateway and AWS lambda, storing the trained model files in S3<br />
<br />
On the other hand, it is expected that there can be some added latency in connecting to the python ML backend server.<br />
<br />
The python backend is exposed as a Flask application. The Flask application is part of the official 'moodlemlbackend' Python package and its FLASK_APP script is 'webapp', in the root of the package. You are free to use the setup that better suits your existing infrastructure.<br />
<br />
====== New server in your infrastructure ======<br />
<br />
The python ML backend is exposed as a Flask application, which uses a WSGI server (https://wsgi.readthedocs.io/en/latest/what.html) to be exposed to the www. The official documentation on how to deploy a Flask app can be found in https://flask.palletsprojects.com/en/1.0.x/tutorial/deploy/.<br />
<br />
* Use MOODLE_MLBACKEND_PYTHON_USERS environment var to set a list of users and password (comma-separated). The value is 'default:sshhhh' (user: default, password: sshhhh).<br />
* Set MOODLE_MLBACKEND_PYTHON_DIR to the path you want to use to store the data generated by the package<br />
<br />
====== Docker ======<br />
<br />
https://hub.docker.com/r/moodlehq/moodle-mlbackend-python is the official moodle-mlbackend-python docker image. We use it internally at Moodle HQ for internal testing and you can use it as well. You may want more control over the image, if that is the case https://github.com/moodlehq/moodle-docker-mlbackend-python/blob/master/Dockerfile can serve as an example of what is needed to get the python moodlemlbackend package working.<br />
<br />
* Use MOODLE_MLBACKEND_PYTHON_USERS environment var to set a list of users and password (comma-separated). The value is 'default:sshhhh' (user: default, password: sshhhh).<br />
<br />
====== AWS serverless service ======<br />
<br />
You can serve the Flask application as a serverless application using the AWS API gateway and AWS lambda. The easiest way to do it is using Zappa https://github.com/Miserlou/Zappa to deploy the Flask application contained in the python ML package.<br />
<br />
Once this has been done, you can select the Python prediction processor as the default or for an individual model:<br />
[[File:python_backend.png|frame|center|Predictions processor default]]<br />
<br />
[[File:python_backend_at_model.png|frame|center|Predictions processor selection for an individual model]]<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
Several analysis intervals are available for models in the system. In this setting, the analysis intervals that will be used [[Using_analytics#Evaluating_models|to evaluate models]] are defined, e.g. so the best analysis interval identified by the evaluation process can be selected for the model. This setting does not restrict the analysis intervals that can be used for specific models.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Defining roles ==<br />
<br />
Moodle learning analytics makes use of a number of capabilities. These can be added or removed from roles at the site level or within certain contexts to customise who can view insights.<br />
<br />
To receive notifications and view insights, a user must have the [[Capabilities/moodle/analytics:listinsights|analytics:listinsights]] capability within the context used as the "Analysable" for the model. For example, the [[Students at risk of dropping out]] model operates within the context of a course. Insights will be generated for each enrolment within any course matching the criteria of the model (courses with a start date in the past and an end date in the future, with at least one teacher and student), and these insights will be sent to anyone with the listinsights capability in that course. By default, the roles of [[Teacher role|Teacher]], [[Non-editing teacher role|Non-editing teacher]], and [[Manager role|Manager]] have this capability.<br />
<br />
Some models (e.g. the ''No teaching'' model) generate insights at the Site level. To receive insights from these models, the user must have a role assignment at the System level which includes the listinsights capability. By default, this is included in the [[Manager role]] if assigned at the site level.<br />
<br />
'''Note''': Site administrators do '''not''' automatically receive insight notifications, though they can choose to view details of any insight notifications on the system. To enable site administrators to receive notifications of insights, assign an additional role that includes the listinsights capability to the site administrator at the system level (e.g. the [[Manager role]]).<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]<br />
[[de:Analytics-Einstellungen]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=136440Analytics settings2020-01-15T03:48:52Z<p>Dmonllao: /* Docker */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
<br />
{{New features}}Analytics may be disabled from Site administration / Advanced features<br />
<br />
They may then be configured from Site administration / Analytics. <br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
==== PHP predictions processor ====<br />
<br />
The PHP processor is the default. There are no other system requirements to use this processor.<br />
<br />
==== Python predictions processor ====<br />
<br />
The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package. The package can be installed in the web server (in all the codes if in a clustered environment) or in a separate server.<br />
<br />
===== Installed in the web server ===== <br />
<br />
The latest version of the package for '''Moodle 3.8''' is compatible with '''Python 3.4, 3.5, 3.6 and 3.7'''. Note that the package should be available for both the Command Line Interface (CLI) user and the user who runs the web server (e.g. www-data).<br />
<br />
# If necessary, install Python 3 (and pip for Python 3)<br />
# Ensure that you use Python 3 to install the moodlemlbackend package:<br />
<br />
sudo -H python3 -m pip install "moodlemlbackend==2.3.*"<br />
# You must also enter the path to the Python 3 executable in Site administration -> Server -> System paths:<br />
[[File:path_to_python_3.png|frame|center|Enter system path for Python 3]]<br />
<br />
===== Installed in a separate server =====<br />
<br />
To install the python package in a separate server instead of installing it on the web server/s have some advantages:<br />
* Keeps the python ML backend as an external service<br />
* To keep a separated control of the resources the web server/s dedicate to serving Moodle and the resources dedicated to the python ML backend<br />
* You can reuse the same ML server for multiple Moodle sites. Easier to setup and maintain than to install/upgrade the python package in all nodes in the cluster<br />
* You can install the package as a new docker container in your dockerized environment<br />
* You can serve the ML backend from AWS through the API gateway and AWS lambda, storing the trained model files in S3<br />
<br />
On the other hand, it is expected that there can be some added latency in connecting to the python ML backend server.<br />
<br />
The python backend is exposed as a Flask application. The Flask application is part of the official 'moodlemlbackend' Python package and its FLASK_APP script is 'webapp', in the root of the package. You are free to use the setup that better suits your existing infrastructure.<br />
<br />
====== New server in your infrastructure ======<br />
<br />
The python ML backend is exposed as a Flask application, which uses a WSGI server (https://wsgi.readthedocs.io/en/latest/what.html) to be exposed to the www. The official documentation on how to deploy a Flask app can be found in https://flask.palletsprojects.com/en/1.0.x/tutorial/deploy/.<br />
<br />
====== Docker ======<br />
<br />
https://hub.docker.com/r/moodlehq/moodle-mlbackend-python is the official moodle-mlbackend-python docker image. We use it internally at Moodle HQ for internal testing and you can use it as well. You may want more control over the image, if that is the case https://github.com/moodlehq/moodle-docker-mlbackend-python/blob/master/Dockerfile can serve as an example of what is needed to get the python moodlemlbackend package working.<br />
<br />
* Use MOODLE_MLBACKEND_PYTHON_USERS environment var to set a list of users and password (comma-separated). The value is 'default:sshhhh' (user: default, password: sshhhh).<br />
<br />
====== AWS serverless service ======<br />
<br />
You can serve the Flask application as a serverless application using the AWS API gateway and AWS lambda. The easiest way to do it is using Zappa https://github.com/Miserlou/Zappa to deploy the Flask application contained in the python ML package.<br />
<br />
Once this has been done, you can select the Python prediction processor as the default or for an individual model:<br />
[[File:python_backend.png|frame|center|Predictions processor default]]<br />
<br />
[[File:python_backend_at_model.png|frame|center|Predictions processor selection for an individual model]]<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
Several analysis intervals are available for models in the system. In this setting, the analysis intervals that will be used [[Using_analytics#Evaluating_models|to evaluate models]] are defined, e.g. so the best analysis interval identified by the evaluation process can be selected for the model. This setting does not restrict the analysis intervals that can be used for specific models.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Defining roles ==<br />
<br />
Moodle learning analytics makes use of a number of capabilities. These can be added or removed from roles at the site level or within certain contexts to customise who can view insights.<br />
<br />
To receive notifications and view insights, a user must have the [[Capabilities/moodle/analytics:listinsights|analytics:listinsights]] capability within the context used as the "Analysable" for the model. For example, the [[Students at risk of dropping out]] model operates within the context of a course. Insights will be generated for each enrolment within any course matching the criteria of the model (courses with a start date in the past and an end date in the future, with at least one teacher and student), and these insights will be sent to anyone with the listinsights capability in that course. By default, the roles of [[Teacher role|Teacher]], [[Non-editing teacher role|Non-editing teacher]], and [[Manager role|Manager]] have this capability.<br />
<br />
Some models (e.g. the ''No teaching'' model) generate insights at the Site level. To receive insights from these models, the user must have a role assignment at the System level which includes the listinsights capability. By default, this is included in the [[Manager role]] if assigned at the site level.<br />
<br />
'''Note''': Site administrators do '''not''' automatically receive insight notifications, though they can choose to view details of any insight notifications on the system. To enable site administrators to receive notifications of insights, assign an additional role that includes the listinsights capability to the site administrator at the system level (e.g. the [[Manager role]]).<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]<br />
[[de:Analytics-Einstellungen]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=136439Analytics settings2020-01-15T03:43:48Z<p>Dmonllao: /* Docker */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
<br />
{{New features}}Analytics may be disabled from Site administration / Advanced features<br />
<br />
They may then be configured from Site administration / Analytics. <br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
==== PHP predictions processor ====<br />
<br />
The PHP processor is the default. There are no other system requirements to use this processor.<br />
<br />
==== Python predictions processor ====<br />
<br />
The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package. The package can be installed in the web server (in all the codes if in a clustered environment) or in a separate server.<br />
<br />
===== Installed in the web server ===== <br />
<br />
The latest version of the package for '''Moodle 3.8''' is compatible with '''Python 3.4, 3.5, 3.6 and 3.7'''. Note that the package should be available for both the Command Line Interface (CLI) user and the user who runs the web server (e.g. www-data).<br />
<br />
# If necessary, install Python 3 (and pip for Python 3)<br />
# Ensure that you use Python 3 to install the moodlemlbackend package:<br />
<br />
sudo -H python3 -m pip install "moodlemlbackend==2.3.*"<br />
# You must also enter the path to the Python 3 executable in Site administration -> Server -> System paths:<br />
[[File:path_to_python_3.png|frame|center|Enter system path for Python 3]]<br />
<br />
===== Installed in a separate server =====<br />
<br />
To install the python package in a separate server instead of installing it on the web server/s have some advantages:<br />
* Keeps the python ML backend as an external service<br />
* To keep a separated control of the resources the web server/s dedicate to serving Moodle and the resources dedicated to the python ML backend<br />
* You can reuse the same ML server for multiple Moodle sites. Easier to setup and maintain than to install/upgrade the python package in all nodes in the cluster<br />
* You can install the package as a new docker container in your dockerized environment<br />
* You can serve the ML backend from AWS through the API gateway and AWS lambda, storing the trained model files in S3<br />
<br />
On the other hand, it is expected that there can be some added latency in connecting to the python ML backend server.<br />
<br />
The python backend is exposed as a Flask application. The Flask application is part of the official 'moodlemlbackend' Python package and its FLASK_APP script is 'webapp', in the root of the package. You are free to use the setup that better suits your existing infrastructure.<br />
<br />
====== New server in your infrastructure ======<br />
<br />
The python ML backend is exposed as a Flask application, which uses a WSGI server (https://wsgi.readthedocs.io/en/latest/what.html) to be exposed to the www. The official documentation on how to deploy a Flask app can be found in https://flask.palletsprojects.com/en/1.0.x/tutorial/deploy/.<br />
<br />
====== Docker ======<br />
<br />
https://hub.docker.com/r/moodlehq/moodle-mlbackend-python is the official moodle-mlbackend-python docker image. We use it internally at Moodle HQ for internal testing and you can use it as well. You may want more control over the image, if that is the case https://github.com/moodlehq/moodle-docker-mlbackend-python/blob/master/Dockerfile can serve as an example of what is needed to get the python moodlemlbackend package working.<br />
<br />
====== AWS serverless service ======<br />
<br />
You can serve the Flask application as a serverless application using the AWS API gateway and AWS lambda. The easiest way to do it is using Zappa https://github.com/Miserlou/Zappa to deploy the Flask application contained in the python ML package.<br />
<br />
Once this has been done, you can select the Python prediction processor as the default or for an individual model:<br />
[[File:python_backend.png|frame|center|Predictions processor default]]<br />
<br />
[[File:python_backend_at_model.png|frame|center|Predictions processor selection for an individual model]]<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
Several analysis intervals are available for models in the system. In this setting, the analysis intervals that will be used [[Using_analytics#Evaluating_models|to evaluate models]] are defined, e.g. so the best analysis interval identified by the evaluation process can be selected for the model. This setting does not restrict the analysis intervals that can be used for specific models.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Defining roles ==<br />
<br />
Moodle learning analytics makes use of a number of capabilities. These can be added or removed from roles at the site level or within certain contexts to customise who can view insights.<br />
<br />
To receive notifications and view insights, a user must have the [[Capabilities/moodle/analytics:listinsights|analytics:listinsights]] capability within the context used as the "Analysable" for the model. For example, the [[Students at risk of dropping out]] model operates within the context of a course. Insights will be generated for each enrolment within any course matching the criteria of the model (courses with a start date in the past and an end date in the future, with at least one teacher and student), and these insights will be sent to anyone with the listinsights capability in that course. By default, the roles of [[Teacher role|Teacher]], [[Non-editing teacher role|Non-editing teacher]], and [[Manager role|Manager]] have this capability.<br />
<br />
Some models (e.g. the ''No teaching'' model) generate insights at the Site level. To receive insights from these models, the user must have a role assignment at the System level which includes the listinsights capability. By default, this is included in the [[Manager role]] if assigned at the site level.<br />
<br />
'''Note''': Site administrators do '''not''' automatically receive insight notifications, though they can choose to view details of any insight notifications on the system. To enable site administrators to receive notifications of insights, assign an additional role that includes the listinsights capability to the site administrator at the system level (e.g. the [[Manager role]]).<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]<br />
[[de:Analytics-Einstellungen]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Using_analytics&diff=136319Using analytics2020-01-02T04:14:36Z<p>Dmonllao: /* Evaluating models */</p>
<hr />
<div>{{Learning analytics}}<br />
== Overview ==<br />
The Moodle Learning Analytics API is an open system that can become the basis for a very wide variety of models. Models can contain indicators (a.k.a. predictors), targets (the outcome we are trying to predict), insights (the predictions themselves), notifications (messages sent as a result of insights), and actions (offered to recipients of messages, which can become indicators in turn). <br />
<br />
[[File:learning_analytics_components.png|thumb]]<br />
<br />
Most learning analytics models are not enabled by default. Enabling models for use should be done after considering the institutional goals the models are meant to support. When selecting or creating an analytics model, the following steps are important:<br />
<br />
* What outcome do we want to predict? Or what process do we want to detect? (Positive or negative)<br />
* How will we detect that outcome/process?<br />
* What clues do we think might help us predict that outcome/process?<br />
* What should we do if the outcome/process is very likely? Very unlikely?<br />
* Who should be notified? What kind of notification should be sent?<br />
* What opportunities for action should be provided on notification?<br />
<br />
Moodle can support multiple prediction models at once, even within the same course. This can be used for A/B testing to compare the performance and accuracy of multiple models. <br />
<br />
Moodle learning analytics supports two types of models. <br />
* '''Machine-learning''' based models, including predictive models, make use of AI models trained using site history to detect or predict hidden aspects of the learning process. <br />
* "'''Static'''" models use a simpler, rule-based system of detecting circumstances on the Moodle site and notifying selected users. <br />
<br />
Moodle core ships with three models, ''[[Students at risk of dropping out]]'' and the static models ''Upcoming activities due'' and ''No teaching.'' Additional prediction models can be created by using the [[dev:Analytics API| Analytics API]] or by using the new web UI. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
You can view and manage your system models from ''Site Administration > Analytics > Analytics models''.<br />
<br />
[[File:manage_models.png|thumb]]<br />
<br />
== Existing models ==<br />
Moodle core ships with three models, [[Students at risk of dropping out]] and the static models Upcoming activities due and No teaching. Other models can be added to your system by installing plugins or by using the web UI (see below). Existing models can be examined and altered from the "Analytics models" page in Site administration:<br />
[[File:analytics_models.png|thumb]]<br />
<br />
These are some of the actions you can perform on an existing model:<br />
<br />
* '''Get predictions''' Train machine learning algorithms with the new data available on the system and get predictions for ongoing courses. ''Predictions are not limited to ongoing courses-- this depends on the model.''<br />
<br />
* '''View Insights''' Once you have trained a machine learning algorithm with the data available on the system, you will see insights (predictions) here for each "analysable." In the included model "[[Students at risk of dropping out]], insights may be selected per course. ''Predictions are not limited to ongoing courses-- this depends on the model.''<br />
<br />
* '''Evaluate''' This is a resource-intensive process, and will not be visible when sites have the "onlycli" setting checked (default). See [https://docs.moodle.org/en/Using_analytics#Triggering_model_evaluation Evaluating models] for more information.<br />
<br />
* '''Log''' View previous evaluation logs, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curve graphs, the tensorboard log dir or the model's Matthews correlation coefficient. The information available will depend on the machine learning backend in use. [[File:log_info.png|thumb]]<br />
<br />
* '''Edit''' You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.<br />
<br />
* '''Enable / Disable''' This allows the model to run training and prediction processes. The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.<br />
<br />
<br />
* '''Export''' Export your site training data to share it with your partner institutions or to use it on a new site. The Export action for models allows you to generate a csv file containing model data about indicators and weights, without exposing any of your site-specific data. We will be asking for submissions of these model files to help evaluate the value of models on different kinds of sites. Please see the [https://moodle.org/project_inspire Learning Analytics community] for more information.<br />
<br />
* '''Invalid site elements''' Reports on what elements in your site can not be analysed by this model<br />
<br />
* '''Clear predictions''' Clears all the model predictions and training data<br />
<br />
=== Core models ===<br />
<br />
==== Students at risk of dropping out ====<br />
<br />
[[Students at risk of dropping out|This model]] predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. In this model, the definition of "dropping out" is "no student activity in the final quarter of the course." The prediction model uses the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry] model of student engagement, consisting of three parts:<br />
<br />
* [[Students at risk of dropping out#Cognitive depth|Cognitive presence]]<br />
* [[Students at risk of dropping out#Social Breadth|Social presence]]<br />
* [[Students at risk of dropping out#Potential indicator levels for selected activity modules|Teacher presence]]<br />
<br />
This prediction model is able to analyse and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success in exact duplicates of courses offered in the past. However, there are some limitations:<br />
<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the [[Students at risk of dropping out#Indicators|indicator]] set (see below). Courses which do not include several core Moodle activities per “time slice” (depending on the time splitting method) will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script:<br />
<br />
$ admin/tool/analytics/cli/guess_course_start_and_end.php <br />
<br />
This script attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
<br />
==== Upcoming activities due ====<br />
<br />
The static “upcoming activities due” model checks for activities with upcoming due dates and outputs to the user’s calendar page.<br />
<br />
==== No teaching ====<br />
This model's insights will inform site managers of which courses with an upcoming start date will not have teaching activity. This is a simple "static" model and it does not use machine learning backend to return predictions. It bases the predictions on assumptions, e.g. there is no teaching if there are no students.<br />
<br />
== Creating and editing models ==<br />
<br />
New machine learning models can be created by using the Analytics API, by importing an exported model from another site, or by using the new web UI. [[File:new_model.png|thumb]] If you delete a "default" model (shipped with Moodle core) you can restore it from the Create menu. (Note: "static" models cannot be created using the web UI at this time.)<br />
<br />
There are four components of a model that can be defined through the web UI:<br />
<br />
=== Target ===<br />
[[File:create_model_2.png|thumb]]Targets represent a “known good”-- something about which we have very strong evidence of value. Targets must be designed carefully to align with the [[Curriculum theory|curriculum priorities]] of the institution. Each model has a single target. The “Analyser” (context in which targets will be evaluated) is automatically controlled by the Target selection. See [[Learning analytics targets]] for more information.<br />
<br />
=== Indicators ===<br />
Indicators are data points that may help to predict targets. We are free to add many indicators to a model to find out if they predict a target-- the only limit is that the data must be available within Moodle and must have a connection to the context of the model (e.g. the user, the course, etc.). The machine learning “training” process will determine how much weight to give to each indicator in the model.<br />
<br />
We do want to make sure any indicators we include in a production model have a clear purpose and can be interpreted by participants, especially if they are used to make prescriptive or diagnostic decisions.<br />
<br />
Indicators are constructed from data, but the data points need to be processed to make consistent, reusable indicators. In many cases, events are counted or combined in some way, though other ways of defining indicators are possible and will be discussed later. How the data points are processed involves important assumptions that affect the indicators. In particular, indicators can be absolute, meaning that the value of the indicator stays the same no matter what other samples are in the context, or relative, meaning that the indicator compares the sample to others in the context.<br />
<br />
See [[Learning analytics indicators]] for more information.<br />
<br />
[[File:indicators.png|thumb]]<br />
<br />
=== Analysis intervals ===<br />
[[File:analysis_intervals.png|thumb]]Analysis intervals control how often the model will run to generate insights, and how much information will be included in each generation cycle. The two analysis intervals enabled by default for selection are “Quarters” and “Quarters accumulative.” Both options will cause models to execute four times-- after the first, second, third and fourth quarters of the course (the final execution is used to evaluate the accuracy of the predictions against the actual outcome). The difference lies in how much information will be included. “Quarters” will only include information from the most recent quarter of the course in its predictions. “Quarters accumulative” will include the most recent quarter and all previous quarters, and tends to generate more accurate predictions (though it can take more time and memory to execute). Moodle Learning Analytics also includes “Tenths” and “Tenths accumulative” options in Core, if you choose to enable them from the Analytics Settings panel. These generate predictions more frequently.<br />
<br />
* '''Single range''' indicates that predictions will be made once, but will take into account a range of time, e.g. one prediction at the end of a course. The prediction is made at the end of the range.<br />
<br />
* '''Upcoming...''' indicates that the model generates an insight based on a snapshot of data at a given moment, e.g. the "no teaching" model looks to see if there are currently any teachers or students assigned to a course one week before the start of the term, and issues one insight warning the site administrator that no teaching is likely to occur in that empty course.<br />
<br />
* '''All previous...''' (formerly "accumulative") and '''Last...''' methods differ in how much data is included in the prediction. Both "All previous quarters" and "Last quarter" predictions are made at the end of each quarter of a time span (e.g. a course), but in "Last quarter," only the information from the most recent quarter is included in the prediction, whereas in "All previous quarters" all information up to the present is included in the prediction.<br />
<br />
'''Single range''' and '''No time splitting''' methods do not have time constraints. They run during the next scheduled task execution, although models apply different restrictions (e.g. require that a course is finished to use it for training or some data in the course and students to use it to get predictions...). 'Single range' and 'No splitting' are not appropriate for students at risk of dropping out of courses. They are intended to be used in models like 'No teaching' or 'Spammer user' that are designed to make only one prediction per eligible sample. To explain this with an example: 'No teaching' model uses 'Single range' analysis interval; the target class (the main PHP class of a model) only accepts courses that will start during the next week. Once we provide a 'No teaching' insight for a course we won't provide any further 'No teaching' insights for that course. <br />
<br />
The difference between 'Single range' and 'No splitting' is that models analysed using 'Single range' will be limited to the analysable elements (the course in students at risk model) start and end dates, while 'No splitting' do not have any time contraints and all data available in the system is used to calculate the indicators.<br />
<br />
'''Note''': Although the examples above refer to courses, analysis intervals can be used on any analysable element. For example, enrolments can have start and end dates, so an analysis interval could be applied to generate predictions about aspects of an enrollment. For analysable elements with no start and end dates, different analysis intervals would be needed. For example, a "weekly" analysis interval could be applied to a model intended to predict whether a user is likely to log in to the system in the future, on the basis of activity in the previous week.<br />
<br />
=== Predictions processor ===<br />
This setting controls which machine learning backend and algorithm will be used to estimate the model. Moodle currently supports two predictions processors:<br />
<br />
* PHP machine learning backend - implements logistic regression using php-ml (contributed by Moodle)<br />
* Python machine learning backend - implements single hidden layer feed-forward neural network using TensorFlow.<br />
<br />
You can only choose from the predictions processors enabled on your site.<br />
<br />
[[File:regression.png|thumb]]<br />
[[File:ffnn.png|thumb]]<br />
<br />
Each prediction processor may support multiple algorithms in the future.<br />
<br />
=== Changing the model name ===<br />
The model name is used to identify insights generated by the model, and by default, is the same as the Target name. You can edit this by clicking the "pencil" icon next to the model name in the list of models:<br />
<br />
[[File:edit_model_name.png|thumb]]<br />
<br />
== Training models ==<br />
Machine-learning based models require a training process using previous data from the site. "Static" models make use of sets of pre-defined rules, and do not need to be trained. <br />
<br />
There are two main categories of machine-learning based analytics models: '''supervised''' and '''unsupervised'''. <br />
<br />
* Supervised models must be trained by using a data set with the target values already identified. For example, if the model will predict course completion, the model must be trained on a set of courses and enrollments with known completion status.<br />
* Unsupervised models look for patterns in existing data, e.g. grouping students based on similarities in their behavior in courses. <br />
<br />
At the present time, Moodle Learning Analytics only supports supervised models.<br />
<br />
While we hope to include pre-trained models with the Moodle core installation in the future, at the current time we do not have large enough data sets to train a model for external use. (If you would like to help contribute data for this effort, please see the [https://moodle.org/course/view.php?id=17233&section=6 Moodle Learning Analytics Working Group].)<br />
<br />
=== Training data ===<br />
The model code includes criteria for "training" and "prediction" data sets. For example, only courses with enrolled students and an end date in the past can be used to train the [[Students at risk of dropping out]] model, because it is impossible to determine whether a student dropped out until a course has ended. On the other hand, for this model to make predictions, there must be a course with students enrolled that has started, but not yet ended.<br />
<br />
The training set is defined in the php code for the Target. Models can only be trained if a site contains enough data matching the training criteria. Most models will require Moodle log data for the time period covering the events being analysed. For example, the [[Students at risk of dropping out]] model can only be trained if there is log data covering student activity in the courses that meet the training criteria. It is possible to train a model on an "archive" system and then use the model on a production system.<br />
<br />
== Evaluating models ==<br />
<br />
[[File:model_menu_evaluate.png|thumb]]This is a '''manual''', resource-intensive process, and will not be visible from the Web UI when sites have the "onlycli" setting checked (default). <br />
<br />
This process can be executed independently of enabling or training the model, and causes Moodle to assemble the training data available on the site, calculate all the indicators and the target and pass the resulting dataset to machine learning backends. This process will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process uses all information available on the site, even if it is very old. Because of this, the accuracy returned by the evaluation process may be lower than the real model accuracy as indicators are more reliably calculated immediately after training data is available because the site state changes over time. The metric used to describe accuracy is a weighted ''[https://en.wikipedia.org/wiki/F1_score F1 score]''.<br />
<br />
''We recommend that all machine-learning models be evaluated before being enabled on a production site.''<br />
<br />
To force the model evaluation process to run from the command line:<br />
<br />
$ admin/tool/analytics/cli/evaluate_model.php<br />
Use the --help option to see parameters.<br />
<br />
If you are connecting to your Moodle server remotely, you will probably want to run the process in the background and detach it, so it will continue to run if you log out of your session (or your connection times out). Here is an example on Linux using bash:<br />
<br />
# php evaluate_model.php --modelid=4 --analysisinterval='\core\analytics\time_splitting\deciles_accum' > eval.log &<br />
# jobs<br />
[1]+ Running php evaluate_model.php --modelid=4 --analysisinterval='\core\analytics\time_splitting\deciles_accum' > eval.log &<br />
# disown -h %1<br />
<br />
This will allow the process to continue to run after the remote shell disconnects. See the documentation for your OS and shell for more details.<br />
<br />
On small sites, you can uncheck "onlycli" in the Analytics settings page, and you can then evaluate models from the Analytics models page. However, this is not feasible with production sites.<br />
<br />
=== Review evaluation results ===<br />
[[File:model_menu_log.png|thumb]]<br />
<br />
You can review the results of the model training process by accessing the evaluation log.[[File:model_log.png|thumb]]<br />
[[File:log_info.png|thumb]]<br />
Check for warnings about evaluation completion, model accuracy, and model variability.<br />
<br />
[[File:invalid_site_elements.png|thumb]]<br />
You can also check the ''invalid site elements'' list to verify which site elements were included or excluded in the analysis. If you see a large number of unexpected elements in this report, it may mean that you need to check your data. For example, if courses don't have appropriate start and end dates set, or enrolment data has been purged, the system may not be able to include data from those courses in the model training process.<br />
<br />
== Exporting and Importing models ==<br />
Models can also be exported from one site and imported to another. [[File:export_menu.png|thumb]] <br />
=== Exporting models ===<br />
You can export the data used to train the model, or the model configuration and the weights of the trained model. [[File:export_dialogue.png|thumb]]<br />
<br />
'''Note''': the model weights are completely anonymous, containing no personally identifiable data! This means it is safe to share them with researchers without worrying about privacy regulations.<br />
<br />
=== Importing models ===<br />
When a model is imported with weights, the site administrator has the option to evaluate the trained model using site data, or to evaluate the model configuration by re-training it using the current site data. [[File:import_model.png|thumb]][[File:evaluate_model.png|thumb]]<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Uso de analítica]]<br />
[[de:Analytics nutzen]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=136228Analytics settings2019-12-16T08:53:55Z<p>Dmonllao: /* Installed in the web server */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
<br />
{{New features}}Analytics may be disabled from Site administration / Advanced features<br />
<br />
They may then be configured from Site administration / Analytics. <br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
==== PHP predictions processor ====<br />
<br />
The PHP processor is the default. There are no other system requirements to use this processor.<br />
<br />
==== Python predictions processor ====<br />
<br />
The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package. The package can be installed in the web server (in all the codes if in a clustered environment) or in a separate server.<br />
<br />
===== Installed in the web server ===== <br />
<br />
The latest version of the package for '''Moodle 3.8''' is compatible with '''Python 3.4, 3.5, 3.6 and 3.7'''. Note that the package should be available for both the Command Line Interface (CLI) user and the user who runs the web server (e.g. www-data).<br />
<br />
# If necessary, install Python 3 (and pip for Python 3)<br />
# Ensure that you use Python 3 to install the moodlemlbackend package:<br />
<br />
sudo -H python3 -m pip install "moodlemlbackend==2.3.*"<br />
# You must also enter the path to the Python 3 executable in Site administration -> Server -> System paths:<br />
[[File:path_to_python_3.png|frame|center|Enter system path for Python 3]]<br />
<br />
===== Installed in a separate server =====<br />
<br />
To install the python package in a separate server instead of installing it on the web server/s have some advantages:<br />
* Keeps the python ML backend as an external service<br />
* To keep a separated control of the resources the web server/s dedicate to serving Moodle and the resources dedicated to the python ML backend<br />
* You can reuse the same ML server for multiple Moodle sites. Easier to setup and maintain than to install/upgrade the python package in all nodes in the cluster<br />
* You can install the package as a new docker container in your dockerized environment<br />
* You can serve the ML backend from AWS through the API gateway and AWS lambda, storing the trained model files in S3<br />
<br />
On the other hand, it is expected that there can be some added latency in connecting to the python ML backend server.<br />
<br />
The python backend is exposed as a Flask application. The Flask application is part of the official 'moodlemlbackend' Python package and its FLASK_APP script is 'webapp', in the root of the package. You are free to use the setup that better suits your existing infrastructure.<br />
<br />
====== New server in your infrastructure ======<br />
<br />
The python ML backend is exposed as a Flask application, which uses a WSGI server (https://wsgi.readthedocs.io/en/latest/what.html) to be exposed to the www. The official documentation on how to deploy a Flask app can be found in https://flask.palletsprojects.com/en/1.0.x/tutorial/deploy/.<br />
<br />
====== Docker ======<br />
<br />
https://hub.docker.com/repository/docker/moodlehq/moodle-mlbackend-python is the official moodle-mlbackend-python docker image. We use it internally at Moodle HQ for internal testing and you can use it as well. You may want more control over the image, if that is the case https://github.com/moodlehq/moodle-docker-mlbackend-python/blob/master/Dockerfile can serve as an example of what is needed to get the package working. <br />
<br />
====== AWS serverless service ======<br />
<br />
You can serve the Flask application as a serverless application using the AWS API gateway and AWS lambda. The easiest way to do it is using Zappa https://github.com/Miserlou/Zappa to deploy the Flask application contained in the python ML package.<br />
<br />
Once this has been done, you can select the Python prediction processor as the default or for an individual model:<br />
[[File:python_backend.png|frame|center|Predictions processor default]]<br />
<br />
[[File:python_backend_at_model.png|frame|center|Predictions processor selection for an individual model]]<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
Several analysis intervals are available for models in the system. In this setting, the analysis intervals that will be used [[Using_analytics#Evaluating_models|to evaluate models]] are defined, e.g. so the best analysis interval identified by the evaluation process can be selected for the model. This setting does not restrict the analysis intervals that can be used for specific models.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Defining roles ==<br />
<br />
Moodle learning analytics makes use of a number of capabilities. These can be added or removed from roles at the site level or within certain contexts to customise who can view insights.<br />
<br />
To receive notifications and view insights, a user must have the [[Capabilities/moodle/analytics:listinsights|analytics:listinsights]] capability within the context used as the "Analysable" for the model. For example, the [[Students at risk of dropping out]] model operates within the context of a course. Insights will be generated for each enrolment within any course matching the criteria of the model (courses with a start date in the past and an end date in the future, with at least one teacher and student), and these insights will be sent to anyone with the listinsights capability in that course. By default, the roles of [[Teacher role|Teacher]], [[Non-editing teacher role|Non-editing teacher]], and [[Manager role|Manager]] have this capability.<br />
<br />
Some models (e.g. the ''No teaching'' model) generate insights at the Site level. To receive insights from these models, the user must have a role assignment at the System level which includes the listinsights capability. By default, this is included in the [[Manager role]] if assigned at the site level.<br />
<br />
'''Note''': Site administrators do '''not''' automatically receive insight notifications, though they can choose to view details of any insight notifications on the system. To enable site administrators to receive notifications of insights, assign an additional role that includes the listinsights capability to the site administrator at the system level (e.g. the [[Manager role]]).<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]<br />
[[de:Analytics-Einstellungen]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=136227Analytics settings2019-12-16T08:50:22Z<p>Dmonllao: /* Predictions processor */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
<br />
{{New features}}Analytics may be disabled from Site administration / Advanced features<br />
<br />
They may then be configured from Site administration / Analytics. <br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
==== PHP predictions processor ====<br />
<br />
The PHP processor is the default. There are no other system requirements to use this processor.<br />
<br />
==== Python predictions processor ====<br />
<br />
The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package. The package can be installed in the web server (in all the codes if in a clustered environment) or in a separate server.<br />
<br />
===== Installed in the web server ===== <br />
<br />
The latest version of the package for '''Moodle 3.8''' is compatible with '''Python 3.4, 3.5, 3.6 and 3.7'''. Note that the package should be available for both the Command Line Interface (CLI) user and the user who runs the web server (e.g. www-data).<br />
<br />
# If necessary, install Python 3 (and pip for Python 3)<br />
# Ensure that you use Python 3 to install the moodlemlbackend package:<br />
<br />
sudo -H python3 -m pip install moodlemlbackend<br />
# You must also enter the path to the Python 3 executable in Site administration -> Server -> System paths:<br />
[[File:path_to_python_3.png|frame|center|Enter system path for Python 3]]<br />
<br />
===== Installed in a separate server =====<br />
<br />
To install the python package in a separate server instead of installing it on the web server/s have some advantages:<br />
* Keeps the python ML backend as an external service<br />
* To keep a separated control of the resources the web server/s dedicate to serving Moodle and the resources dedicated to the python ML backend<br />
* You can reuse the same ML server for multiple Moodle sites. Easier to setup and maintain than to install/upgrade the python package in all nodes in the cluster<br />
* You can install the package as a new docker container in your dockerized environment<br />
* You can serve the ML backend from AWS through the API gateway and AWS lambda, storing the trained model files in S3<br />
<br />
On the other hand, it is expected that there can be some added latency in connecting to the python ML backend server.<br />
<br />
The python backend is exposed as a Flask application. The Flask application is part of the official 'moodlemlbackend' Python package and its FLASK_APP script is 'webapp', in the root of the package. You are free to use the setup that better suits your existing infrastructure.<br />
<br />
====== New server in your infrastructure ======<br />
<br />
The python ML backend is exposed as a Flask application, which uses a WSGI server (https://wsgi.readthedocs.io/en/latest/what.html) to be exposed to the www. The official documentation on how to deploy a Flask app can be found in https://flask.palletsprojects.com/en/1.0.x/tutorial/deploy/.<br />
<br />
====== Docker ======<br />
<br />
https://hub.docker.com/repository/docker/moodlehq/moodle-mlbackend-python is the official moodle-mlbackend-python docker image. We use it internally at Moodle HQ for internal testing and you can use it as well. You may want more control over the image, if that is the case https://github.com/moodlehq/moodle-docker-mlbackend-python/blob/master/Dockerfile can serve as an example of what is needed to get the package working. <br />
<br />
====== AWS serverless service ======<br />
<br />
You can serve the Flask application as a serverless application using the AWS API gateway and AWS lambda. The easiest way to do it is using Zappa https://github.com/Miserlou/Zappa to deploy the Flask application contained in the python ML package.<br />
<br />
Once this has been done, you can select the Python prediction processor as the default or for an individual model:<br />
[[File:python_backend.png|frame|center|Predictions processor default]]<br />
<br />
[[File:python_backend_at_model.png|frame|center|Predictions processor selection for an individual model]]<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
Several analysis intervals are available for models in the system. In this setting, the analysis intervals that will be used [[Using_analytics#Evaluating_models|to evaluate models]] are defined, e.g. so the best analysis interval identified by the evaluation process can be selected for the model. This setting does not restrict the analysis intervals that can be used for specific models.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Defining roles ==<br />
<br />
Moodle learning analytics makes use of a number of capabilities. These can be added or removed from roles at the site level or within certain contexts to customise who can view insights.<br />
<br />
To receive notifications and view insights, a user must have the [[Capabilities/moodle/analytics:listinsights|analytics:listinsights]] capability within the context used as the "Analysable" for the model. For example, the [[Students at risk of dropping out]] model operates within the context of a course. Insights will be generated for each enrolment within any course matching the criteria of the model (courses with a start date in the past and an end date in the future, with at least one teacher and student), and these insights will be sent to anyone with the listinsights capability in that course. By default, the roles of [[Teacher role|Teacher]], [[Non-editing teacher role|Non-editing teacher]], and [[Manager role|Manager]] have this capability.<br />
<br />
Some models (e.g. the ''No teaching'' model) generate insights at the Site level. To receive insights from these models, the user must have a role assignment at the System level which includes the listinsights capability. By default, this is included in the [[Manager role]] if assigned at the site level.<br />
<br />
'''Note''': Site administrators do '''not''' automatically receive insight notifications, though they can choose to view details of any insight notifications on the system. To enable site administrators to receive notifications of insights, assign an additional role that includes the listinsights capability to the site administrator at the system level (e.g. the [[Manager role]]).<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]<br />
[[de:Analytics-Einstellungen]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=135129Analytics settings2019-08-26T07:53:32Z<p>Dmonllao: /* Predictions processor */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
{{New features}}<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
[[File:configure_settings.png|thumb]] The settings for the Moodle Learning Analytics system are set to reasonable defaults, but let’s review them. To configure and enable Moodle Learning Analytics settings, access the Analytics settings panel under Site Administration/Analytics.<br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package. The latest version of the package for Moodle 3.7 is compatible with python 2.7 and python 3.4. Note that the package should be available for both the Command Line Interface (CLI) user and the user who runs the web server (e.g. www-data).<br />
<br />
pip install "moodlemlbackend>=1.0.0,<2.0.0"<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Defining roles ==<br />
<br />
Moodle learning analytics makes use of a number of capabilities. These can be added or removed from roles at the site level or within certain contexts to customise who can view insights.<br />
<br />
To receive notifications and view insights, a user must have the [[Capabilities/moodle/analytics:listinsights|analytics:listinsights]] capability within the context used as the "Analysable" for the model. For example, the [[Students at risk of dropping out]] model operates within the context of a course. Insights will be generated for each enrolment within any course matching the criteria of the model (courses with a start date in the past and an end date in the future, with at least one teacher and student), and these insights will be sent to anyone with the listinsights capability in that course. By default, the roles of [[Teacher role|Teacher]], [[Non-editing teacher role|Non-editing teacher]], and [[Manager role|Manager]] have this capability.<br />
<br />
Some models (e.g. the ''No teaching'' model) generate insights at the Site level. To receive insights from these models, the user must have a role assignment at the System level which includes the listinsights capability. By default, this is included in the [[Manager role]] if assigned at the site level.<br />
<br />
'''Note''': Site administrators do '''not''' automatically receive insight notifications, though they can choose to view details of any insight notifications on the system. To enable site administrators to receive notifications of insights, assign an additional role that includes the listinsights capability to the site administrator at the system level (e.g. the [[Manager role]]).<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]<br />
[[de:Analytics-Einstellungen]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=134740Analytics settings2019-07-10T11:35:10Z<p>Dmonllao: /* Predictions processor */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
{{New features}}<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
[[File:configure_settings.png|thumb]] The settings for the Moodle Learning Analytics system are set to reasonable defaults, but let’s review them. To configure and enable Moodle Learning Analytics settings, access the Analytics settings panel under Site Administration/Analytics.<br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package. The latest version of the package for Moodle 3.7 is compatible with python 2.7 and python 3.4.<br />
<br />
pip install "moodlemlbackend>=1.0.0,<2.0.0"<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics_settings&diff=134739Analytics settings2019-07-10T11:33:19Z<p>Dmonllao: /* Predictions processor */</p>
<hr />
<div>{{Learning analytics}}<br />
The Moodle learning analytics system requires some initial configuration before it can be used. You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
== Site information ==<br />
{{New features}}<br />
Site information will be used to help learning analytics models take characteristics of the institution into account. This information is also reported as part of site data collection when you register your site. This will allow HQ to understand which areas in learning analytics are seeing the most use and prioritize development resources appropriately.<br />
<br />
== Configure learning analytics settings ==<br />
[[File:configure_settings.png|thumb]] The settings for the Moodle Learning Analytics system are set to reasonable defaults, but let’s review them. To configure and enable Moodle Learning Analytics settings, access the Analytics settings panel under Site Administration/Analytics.<br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python processor is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package.<br />
<br />
pip install "moodlemlbackend>=1.0.0,<2.0.0"<br />
<br />
=== Log store ===<br />
From Moodle version 2.7 and up, the “Standard logstore” is the default. If for some reason you also have data in the older “legacy logs,” you can enable the Moodle Learning Analytics system to access them instead.<br />
<br />
=== Analysis intervals ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Analysis intervals]] determine how often insights will be generated, and how much information to use for each calculation. Using proportional analysis intervals allows courses of different lengths to be used to train a single model.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Analysis intervals]]<br />
<br />
Each analysis interval divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the analysis intervals you are interested in using; the evaluation process will iterate through all enabled analysis intervals, so the more analysis intervals enabled, the slower the evaluation process will be.<br />
<br />
=== Models output directory ===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
== Scheduled tasks ==<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Configuraciones de analítica]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics&diff=133076Analytics2019-01-29T20:26:50Z<p>Dmonllao: /* Model management */</p>
<hr />
<div>{{Tracking progress}}<br />
==Overview==<br />
<br />
{{MediaPlayer | url = https://youtu.be/MS1IqKsrXAI | desc = Analytics overview}}<br />
<br />
Beginning in version 3.4, Moodle core now implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.<br />
<br />
In Moodle 3.4, this system ships with two built-in models:<br />
<br />
* [[Students at risk of dropping out]]<br />
* No teaching activity<br />
<br />
The system can be easily extended with new custom models, based on reusable targets, indicators, and other components. For more information, see the [[dev:Analytics API| Analytics API]] developer documentation.<br />
<br />
===Features===<br />
* Two built-in prediction models: "[[Students at risk of dropping out]]" and "No Teaching".<br />
* A set of student engagement indicators based on the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry].<br />
* Built-in tools to evaluate models against your site's data<br />
* Proactive notifications for instructors using Events<br />
* Instructors can easily send messages to students identified by the model, or jump to the Outline report for that student for more detail about student activity<br />
* An [[dev:Analytics API| API]] to build indicators and prediction models for third-party Moodle plugins<br />
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends<br />
<br />
'''Note: PHP 7.x is required.'''<br />
<br />
===Limitations===<br />
This release of Moodle Learning Analytics has the following limitations:<br />
<br />
* Models included in this release must be "trained" on a site with previous completed courses, ideally using the Moodle course completion feature. The current models cannot make predictions on a site until this is done.<br />
* The prediction model included with this version requires that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle.<br />
* Models and predictions are only visible to teachers and administrators at present.<br />
<br />
We are continuing to enhance Moodle Learning Analytics, and expanded capabilities will be released going forward. To help contribute to our progress, please join the conversation at the [https://moodle.org/project_inspire Moodle Learning Analytics Community]. In particular, we still need data sets from a wide variety of Moodle-using institutions in order to be able to ship a working prediction model that does not depend on local site data before it can be used.<br />
<br />
== Settings ==<br />
<br />
You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python one is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package.<br />
<br />
pip install moodlemlbackend<br />
<br />
=== Time splitting methods ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Time splitting methods]] allow insights generated from one course to be used on another course, even if the two courses are not exactly the same length.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Time splitting methods]]<br />
<br />
Each time splitting method divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the time splitting methods you are interested in using; the evaluation process will iterate through all enabled time-spitting methods, so the more time-splitting methods enabled, the slower the evaluation process will be.<br />
<br />
"Single range" indicates that predictions will be made once, but will take into account a range of time, e.g. one prediction at the end of a course. The prediction is made at the end of the range.<br />
<br />
"No splitting" indicates that the model generates an insight based on a snapshot of data at a given moment, e.g. the "no teaching" model looks to see if there are currently any teachers or students assigned to a course at a defined point before the start of the term, and issues one insight warning the site administrator that no teaching is likely to occur in that empty course.<br />
<br />
"Accumulative" methods differ in how much data is included in the prediction. Both "quarterly" and "quarterly accumulative" predictions are made at the end of each quarter of a time span (e.g. a course), but in "quarterly," only the information from the most recent quarter is included in the prediction, whereas in "quarterly accumulative" all information up to the present is included in the prediction.<br />
<br />
Single range and no splitting methods do not have time constraints. They run during the next scheduled task execution, although models apply different restrictions (e.g. require that a course is finished to use it for training or some data in the course and students to use it to get predictions...). 'Single range' and 'No splitting' are not appropriate for students at risk of dropping out of courses. They are intended to be used in models like 'No teaching' or 'Spammer user' where you just want one prediction and done. To explain this with an example: 'No teaching' model uses 'Single range' time splitting method; the target class (the main PHP class of a model) only accepts courses that will start during the next week. Once we provide 'No teaching' insights for a course we don't provide them anymore. <br />
<br />
The difference between 'Single range' and 'No splitting' is that models analysed using 'Single range' will be limited to the analysable elements (the course in students at risk model) start and end dates, while 'No splitting' do not have any time contraints and all data available in the system is used to calculate the indicators.<br />
<br />
'''Note''': Although the examples above refer to courses, time splitting methods can be used on any analysable element. For example, enrolments can have start and end dates, so a time splitting method could be applied to generate predictions about aspects of an enrollment. For analysable elements with no start and end dates, different time splitting methods would be needed. For example, a "weekly" time splitting method could be applied to a model intended to predict whether a user is likely to log in to the system in the future, on the basis of activity in the previous week.<br />
<br />
===Models output directory===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
===Scheduled tasks===<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Model management ==<br />
<br />
Moodle can support multiple prediction models at once, even within the same course. This can be used for A/B testing to compare the performance and accuracy of multiple models. <br />
<br />
Moodle core ships with two prediction models, ''[[Students at risk of dropping out]]'' and ''No teaching.'' Additional prediction models can be created by using the [[dev:Analytics API| Analytics API]]. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
You can manage your system models from ''Site Administration > Analytics > Analytics models''.<br />
<br />
[[File:prediction-models-list.jpeg]]<br />
<br />
These are some of the actions you can perform on a model:<br />
<br />
=== View Insights ===<br />
Once you have trained a machine learning algorithm with the data available on the system, you will see insights (predictions) here for each "analysable." In the included model "[[Students at risk of dropping out]], insights may be selected per course. ''Predictions are not limited to ongoing courses-- this depends on the model.''<br />
<br />
=== Evaluate ===<br />
Evaluate the prediction model by getting all the training data available on the site, calculating all the indicators and the target and passing the resulting dataset to machine learning backends. This process will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process uses all information available on the site, even if it is very old. Because of this, the accuracy returned by the evaluation process may be lower than the real model accuracy as indicators are more reliably calculated immediately after training data is available because the site state changes over time. The metric used to describe accuracy is the ''[https://en.wikipedia.org/wiki/Matthews_correlation_coefficient Matthews correlation coefficient]'' (a metric used in machine learning for evaluating binary classifications)<br />
<br />
You can force the model evaluation process to run from the command line:<br />
<br />
$ admin/tool/analytics/cli/evaluate_model.php<br />
<br />
[[File:model-evaluation.jpeg]]<br />
<br />
=== Log ===<br />
View previous evaluation logs, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curve graphs, the tensorboard log dir or the model's Matthews correlation coefficient. The information available will depend on the machine learning backend in use.<br />
<br />
=== Edit ===<br />
You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.<br />
<br />
=== Enable / Disable ===<br />
The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.<br />
<br />
=== Export ===<br />
Export your site training data to share it with your partner institutions or to use it on a new site. The Export action for models allows you to generate a csv file containing model data about indicators and weights, without exposing any of your site-specific data. We will be asking for submissions of these model files to help evaluate the value of models on different kinds of sites. Please see the [https://moodle.org/project_inspire Learning Analytics community] for more information.<br />
<br />
=== Invalid site elements ===<br />
Reports on what elements in your site can not be analysed by this model<br />
=== Clear predictions ===<br />
Clears all the model predictions and training data<br />
<br />
=== Core models ===<br />
<br />
==== Students at risk of dropping out ====<br />
<br />
[[Students at risk of dropping out|This model]] predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. In this model, the definition of "dropping out" is "no student activity in the final quarter of the course." The prediction model uses the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry] model of student engagement, consisting of three parts:<br />
<br />
* [[Students at risk of dropping out#Cognitive depth|Cognitive presence]]<br />
* [[Students at risk of dropping out#Social Breadth|Social presence]]<br />
* [[Students at risk of dropping out#Potential indicator levels for selected activity modules|Teacher presence]]<br />
<br />
This prediction model is able to analyse and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success in exact duplicates of courses offered in the past. However, there are some limitations:<br />
<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the [[Students at risk of dropping out#Indicators|indicator]] set (see below). Courses which do not include several core Moodle activities per “time slice” (depending on the time splitting method) will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script:<br />
<br />
$ admin/tool/analytics/cli/guess_course_start_and_end.php <br />
<br />
This script attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
<br />
==== No teaching ====<br />
<br />
This model's insights will inform site managers of which courses with an upcoming start date will not have teaching activity. This is a simple model and it does not use machine learning backend to return predictions. It bases the predictions on assumptions, e.g. there is no teaching if there are no students.<br />
<br />
== Predictions and Insights ==<br />
<br />
Models will start generating predictions at different points in time, depending on the site prediction models and the site courses start and end dates. <br />
<br />
Each model defines which predictions will generate [[dev:Analytics API#Insights|insights]] and which predictions will be ignored. For example, the ''[[Students at risk of dropping out]]'' prediction model does not generate an insight if a student is predicted as "not at risk," since the primary interest is which students are at risk of dropping out of courses, not which students are not at risk.<br />
<br />
[[File:prediction-model-insights.jpeg]]<br />
<br />
Users can specify how they wish to receive insights notifications, or turn them off, via their User menu > Preferences > Notification preferences.<br />
<br />
=== Actions===<br />
<br />
Each insight can have one or more actions defined. Actions provide a way to act on the insight as it is read. These actions may include a way to send a message to another user, a link to a report providing information about the sample the prediction has been generated for (e.g. a report for an existing student), or a way to view the details of the model prediction.<br />
<br />
[[Image:08_insights.png|frame|center|Insights and Actions]]<br />
<br />
Insights can also offer two important general actions that are applicable to all insights. First, the user can acknowledge the insight. This removes that particular prediction from the view of the user, e.g. a notification about a particular student at risk is removed from the display.<br />
<br />
The second general action is to mark the insight as "Not useful." This also removes the insight associated with this calculation from the display, but the model is adjusted to make this prediction less likely in the future.<br />
<br />
==Capabilities==<br />
<br />
There are two analytics capabilities:<br />
<br />
* [[Capabilities/moodle/analytics:managemodels|Manage models]] - allowed for the default role of manager only<br />
* [[Capabilities/moodle/analytics:listinsights|List insights]] - allowed for the default roles of manager, teacher and non-editing teacher<br />
<br />
<br />
==Frequently Asked Questions==<br />
[[Moodle Learning Analytics FAQ]]<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Analítica]]<br />
[[de:Analytics]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics&diff=133075Analytics2019-01-29T20:09:45Z<p>Dmonllao: /* Evaluate */</p>
<hr />
<div>{{Tracking progress}}<br />
==Overview==<br />
<br />
{{MediaPlayer | url = https://youtu.be/MS1IqKsrXAI | desc = Analytics overview}}<br />
<br />
Beginning in version 3.4, Moodle core now implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.<br />
<br />
In Moodle 3.4, this system ships with two built-in models:<br />
<br />
* [[Students at risk of dropping out]]<br />
* No teaching activity<br />
<br />
The system can be easily extended with new custom models, based on reusable targets, indicators, and other components. For more information, see the [[dev:Analytics API| Analytics API]] developer documentation.<br />
<br />
===Features===<br />
* Two built-in prediction models: "[[Students at risk of dropping out]]" and "No Teaching".<br />
* A set of student engagement indicators based on the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry].<br />
* Built-in tools to evaluate models against your site's data<br />
* Proactive notifications for instructors using Events<br />
* Instructors can easily send messages to students identified by the model, or jump to the Outline report for that student for more detail about student activity<br />
* An [[dev:Analytics API| API]] to build indicators and prediction models for third-party Moodle plugins<br />
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends<br />
<br />
'''Note: PHP 7.x is required.'''<br />
<br />
===Limitations===<br />
This release of Moodle Learning Analytics has the following limitations:<br />
<br />
* Models included in this release must be "trained" on a site with previous completed courses, ideally using the Moodle course completion feature. The current models cannot make predictions on a site until this is done.<br />
* The prediction model included with this version requires that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle.<br />
* Models and predictions are only visible to teachers and administrators at present.<br />
<br />
We are continuing to enhance Moodle Learning Analytics, and expanded capabilities will be released going forward. To help contribute to our progress, please join the conversation at the [https://moodle.org/project_inspire Moodle Learning Analytics Community]. In particular, we still need data sets from a wide variety of Moodle-using institutions in order to be able to ship a working prediction model that does not depend on local site data before it can be used.<br />
<br />
== Settings ==<br />
<br />
You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python one is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package.<br />
<br />
pip install moodlemlbackend<br />
<br />
=== Time splitting methods ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Time splitting methods]] allow insights generated from one course to be used on another course, even if the two courses are not exactly the same length.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Time splitting methods]]<br />
<br />
Each time splitting method divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the time splitting methods you are interested in using; the evaluation process will iterate through all enabled time-spitting methods, so the more time-splitting methods enabled, the slower the evaluation process will be.<br />
<br />
"Single range" indicates that predictions will be made once, but will take into account a range of time, e.g. one prediction at the end of a course. The prediction is made at the end of the range.<br />
<br />
"No splitting" indicates that the model generates an insight based on a snapshot of data at a given moment, e.g. the "no teaching" model looks to see if there are currently any teachers or students assigned to a course at a defined point before the start of the term, and issues one insight warning the site administrator that no teaching is likely to occur in that empty course.<br />
<br />
"Accumulative" methods differ in how much data is included in the prediction. Both "quarterly" and "quarterly accumulative" predictions are made at the end of each quarter of a time span (e.g. a course), but in "quarterly," only the information from the most recent quarter is included in the prediction, whereas in "quarterly accumulative" all information up to the present is included in the prediction.<br />
<br />
Single range and no splitting methods do not have time constraints. They run during the next scheduled task execution, although models apply different restrictions (e.g. require that a course is finished to use it for training or some data in the course and students to use it to get predictions...). 'Single range' and 'No splitting' are not appropriate for students at risk of dropping out of courses. They are intended to be used in models like 'No teaching' or 'Spammer user' where you just want one prediction and done. To explain this with an example: 'No teaching' model uses 'Single range' time splitting method; the target class (the main PHP class of a model) only accepts courses that will start during the next week. Once we provide 'No teaching' insights for a course we don't provide them anymore. <br />
<br />
The difference between 'Single range' and 'No splitting' is that models analysed using 'Single range' will be limited to the analysable elements (the course in students at risk model) start and end dates, while 'No splitting' do not have any time contraints and all data available in the system is used to calculate the indicators.<br />
<br />
'''Note''': Although the examples above refer to courses, time splitting methods can be used on any analysable element. For example, enrolments can have start and end dates, so a time splitting method could be applied to generate predictions about aspects of an enrollment. For analysable elements with no start and end dates, different time splitting methods would be needed. For example, a "weekly" time splitting method could be applied to a model intended to predict whether a user is likely to log in to the system in the future, on the basis of activity in the previous week.<br />
<br />
===Models output directory===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
===Scheduled tasks===<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Model management ==<br />
<br />
Moodle can support multiple prediction models at once, even within the same course. This can be used for A/B testing to compare the performance and accuracy of multiple models. <br />
<br />
Moodle core ships with two prediction models, ''[[Students at risk of dropping out]]'' and ''No teaching.'' Additional prediction models can be created by using the [[dev:Analytics API| Analytics API]]. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
You can manage your system models from ''Site Administration > Analytics > Analytics models''.<br />
<br />
[[File:prediction-models-list.jpeg]]<br />
<br />
These are some of the actions you can perform on a model:<br />
<br />
=== View Insights ===<br />
Once you have trained a machine learning algorithm with the data available on the system, you will see insights (predictions) here for each "analysable." In the included model "[[Students at risk of dropping out]], insights may be selected per course. ''Predictions are not limited to ongoing courses-- this depends on the model.''<br />
<br />
=== Evaluate ===<br />
Evaluate the prediction model by getting all the training data available on the site, calculating all the indicators and the target and passing the resulting dataset to machine learning backends. This process will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process uses all information available on the site, even if it is very old. Because of this, the accuracy returned by the evaluation process may be lower than the real model accuracy as indicators are more reliably calculated immediately after training data is available because the site state changes over time. The metric used to describe accuracy is the ''[https://en.wikipedia.org/wiki/Matthews_correlation_coefficient Matthews correlation coefficient]'' (a metric used in machine learning for evaluating binary classifications)<br />
<br />
You can force the model evaluation process to run from the command line:<br />
<br />
$ admin/tool/analytics/cli/evaluate_model.php<br />
<br />
=== Log ===<br />
View previous evaluation logs, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curve graphs, the tensorboard log dir or the model's Matthews correlation coefficient. The information available will depend on the machine learning backend in use.<br />
<br />
=== Edit ===<br />
You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.<br />
<br />
=== Enable / Disable ===<br />
The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.<br />
<br />
=== Export ===<br />
Export your site training data to share it with your partner institutions or to use it on a new site. The Export action for models allows you to generate a csv file containing model data about indicators and weights, without exposing any of your site-specific data. We will be asking for submissions of these model files to help evaluate the value of models on different kinds of sites. Please see the [https://moodle.org/project_inspire Learning Analytics community] for more information.<br />
<br />
=== Invalid site elements ===<br />
Reports on what elements in your site can not be analysed by this model<br />
=== Clear predictions ===<br />
Clears all the model predictions and training data<br />
<br />
[[File:model-evaluation.jpeg]]<br />
<br />
<br />
=== Core models ===<br />
<br />
==== Students at risk of dropping out ====<br />
<br />
[[Students at risk of dropping out|This model]] predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. In this model, the definition of "dropping out" is "no student activity in the final quarter of the course." The prediction model uses the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry] model of student engagement, consisting of three parts:<br />
<br />
* [[Students at risk of dropping out#Cognitive depth|Cognitive presence]]<br />
* [[Students at risk of dropping out#Social Breadth|Social presence]]<br />
* [[Students at risk of dropping out#Potential indicator levels for selected activity modules|Teacher presence]]<br />
<br />
This prediction model is able to analyse and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success in exact duplicates of courses offered in the past. However, there are some limitations:<br />
<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the [[Students at risk of dropping out#Indicators|indicator]] set (see below). Courses which do not include several core Moodle activities per “time slice” (depending on the time splitting method) will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script:<br />
<br />
$ admin/tool/analytics/cli/guess_course_start_and_end.php <br />
<br />
This script attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
<br />
==== No teaching ====<br />
<br />
This model's insights will inform site managers of which courses with an upcoming start date will not have teaching activity. This is a simple model and it does not use machine learning backend to return predictions. It bases the predictions on assumptions, e.g. there is no teaching if there are no students.<br />
<br />
== Predictions and Insights ==<br />
<br />
Models will start generating predictions at different points in time, depending on the site prediction models and the site courses start and end dates. <br />
<br />
Each model defines which predictions will generate [[dev:Analytics API#Insights|insights]] and which predictions will be ignored. For example, the ''[[Students at risk of dropping out]]'' prediction model does not generate an insight if a student is predicted as "not at risk," since the primary interest is which students are at risk of dropping out of courses, not which students are not at risk.<br />
<br />
[[File:prediction-model-insights.jpeg]]<br />
<br />
Users can specify how they wish to receive insights notifications, or turn them off, via their User menu > Preferences > Notification preferences.<br />
<br />
=== Actions===<br />
<br />
Each insight can have one or more actions defined. Actions provide a way to act on the insight as it is read. These actions may include a way to send a message to another user, a link to a report providing information about the sample the prediction has been generated for (e.g. a report for an existing student), or a way to view the details of the model prediction.<br />
<br />
[[Image:08_insights.png|frame|center|Insights and Actions]]<br />
<br />
Insights can also offer two important general actions that are applicable to all insights. First, the user can acknowledge the insight. This removes that particular prediction from the view of the user, e.g. a notification about a particular student at risk is removed from the display.<br />
<br />
The second general action is to mark the insight as "Not useful." This also removes the insight associated with this calculation from the display, but the model is adjusted to make this prediction less likely in the future.<br />
<br />
==Capabilities==<br />
<br />
There are two analytics capabilities:<br />
<br />
* [[Capabilities/moodle/analytics:managemodels|Manage models]] - allowed for the default role of manager only<br />
* [[Capabilities/moodle/analytics:listinsights|List insights]] - allowed for the default roles of manager, teacher and non-editing teacher<br />
<br />
<br />
==Frequently Asked Questions==<br />
[[Moodle Learning Analytics FAQ]]<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Analítica]]<br />
[[de:Analytics]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics&diff=132959Analytics2019-01-14T17:41:03Z<p>Dmonllao: /* Models output directory */</p>
<hr />
<div>{{Tracking progress}}<br />
==Overview==<br />
<br />
{{MediaPlayer | url = https://youtu.be/MS1IqKsrXAI | desc = Analytics overview}}<br />
<br />
Beginning in version 3.4, Moodle core now implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.<br />
<br />
In Moodle 3.4, this system ships with two built-in models:<br />
<br />
* [[Students at risk of dropping out]]<br />
* No teaching activity<br />
<br />
The system can be easily extended with new custom models, based on reusable targets, indicators, and other components. For more information, see the [[dev:Analytics API| Analytics API]] developer documentation.<br />
<br />
===Features===<br />
* Two built-in prediction models: "[[Students at risk of dropping out]]" and "No Teaching".<br />
* A set of student engagement indicators based on the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry].<br />
* Built-in tools to evaluate models against your site's data<br />
* Proactive notifications for instructors using Events<br />
* Instructors can easily send messages to students identified by the model, or jump to the Outline report for that student for more detail about student activity<br />
* An [[dev:Analytics API| API]] to build indicators and prediction models for third-party Moodle plugins<br />
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends<br />
<br />
'''Note: PHP 7.x is required.'''<br />
<br />
===Limitations===<br />
This release of Moodle Learning Analytics has the following limitations:<br />
<br />
* Models included in this release must be "trained" on a site with previous completed courses, ideally using the Moodle course completion feature. The current models cannot make predictions on a site until this is done.<br />
* The prediction model included with this version requires that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle.<br />
* Models and predictions are only visible to teachers and administrators at present.<br />
<br />
We are continuing to enhance Moodle Learning Analytics, and expanded capabilities will be released going forward. To help contribute to our progress, please join the conversation at the [https://moodle.org/project_inspire Moodle Learning Analytics Community]. In particular, we still need data sets from a wide variety of Moodle-using institutions in order to be able to ship a working prediction model that does not depend on local site data before it can be used.<br />
<br />
== Settings ==<br />
<br />
You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python one is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package.<br />
<br />
pip install moodlemlbackend<br />
<br />
=== Time splitting methods ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Time splitting methods]] allow insights generated from one course to be used on another course, even if the two courses are not exactly the same length.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Time splitting methods]]<br />
<br />
Each time splitting method divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the time splitting methods you are interested in using; the evaluation process will iterate through all enabled time-spitting methods, so the more time-splitting methods enabled, the slower the evaluation process will be.<br />
<br />
"Single range" indicates that predictions will be made once, but will take into account a range of time, e.g. one prediction at the end of a course. The prediction is made at the end of the range.<br />
<br />
"No splitting" indicates that the model generates an insight based on a snapshot of data at a given moment, e.g. the "no teaching" model looks to see if there are currently any teachers or students assigned to a course at a defined point before the start of the term, and issues one insight warning the site administrator that no teaching is likely to occur in that empty course.<br />
<br />
"Accumulative" methods differ in how much data is included in the prediction. Both "quarterly" and "quarterly accumulative" predictions are made at the end of each quarter of a time span (e.g. a course), but in "quarterly," only the information from the most recent quarter is included in the prediction, whereas in "quarterly accumulative" all information up to the present is included in the prediction.<br />
<br />
Single range and no splitting methods do not have time constraints. They run during the next scheduled task execution, although models apply different restrictions (e.g. require that a course is finished to use it for training or some data in the course and students to use it to get predictions...). 'Single range' and 'No splitting' are not appropriate for students at risk of dropping out of courses. They are intended to be used in models like 'No teaching' or 'Spammer user' where you just want one prediction and done. To explain this with an example: 'No teaching' model uses 'Single range' time splitting method; the target class (the main PHP class of a model) only accepts courses that will start during the next week. Once we provide 'No teaching' insights for a course we don't provide them anymore. <br />
<br />
The difference between 'Single range' and 'No splitting' is that models analysed using 'Single range' will be limited to the analysable elements (the course in students at risk model) start and end dates, while 'No splitting' do not have any time contraints and all data available in the system is used to calculate the indicators.<br />
<br />
'''Note''': Although the examples above refer to courses, time splitting methods can be used on any analysable element. For example, enrolments can have start and end dates, so a time splitting method could be applied to generate predictions about aspects of an enrollment. For analysable elements with no start and end dates, different time splitting methods would be needed. For example, a "weekly" time splitting method could be applied to a model intended to predict whether a user is likely to log in to the system in the future, on the basis of activity in the previous week.<br />
<br />
===Models output directory===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server. This setting can be used by Moodle sites with multiple frontend nodes (a cluster) to specify a shared directory across nodes. This directory can be used by machine learning backends to store trained algorithms (its internal variables weights and stuff like that) to use them later to get predictions. Moodle cron lock will prevent multiple executions of the analytics tasks that train machine learning algorithms and get predictions from them.<br />
<br />
===Scheduled tasks===<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Model management ==<br />
<br />
Moodle can support multiple prediction models at once, even within the same course. This can be used for A/B testing to compare the performance and accuracy of multiple models. <br />
<br />
Moodle core ships with two prediction models, ''[[Students at risk of dropping out]]'' and ''No teaching.'' Additional prediction models can be created by using the [[dev:Analytics API| Analytics API]]. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
You can manage your system models from ''Site Administration > Analytics > Analytics models''.<br />
<br />
[[File:prediction-models-list.jpeg]]<br />
<br />
These are some of the actions you can perform on a model:<br />
<br />
=== View Insights ===<br />
Once you have trained a machine learning algorithm with the data available on the system, you will see insights (predictions) here for each "analysable." In the included model "[[Students at risk of dropping out]], insights may be selected per course. ''Predictions are not limited to ongoing courses-- this depends on the model.''<br />
<br />
=== Evaluate ===<br />
(disabled by default) Evaluate the prediction model by getting all the training data available on the site, calculating all the indicators and the target and passing the resulting dataset to machine learning backends. This process will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process uses all information available on the site, even if it is very old. Because of this, the accuracy returned by the evaluation process may be lower than the real model accuracy as indicators are more reliably calculated immediately after training data is available because the site state changes over time. The metric used to describe accuracy is the ''[https://en.wikipedia.org/wiki/Matthews_correlation_coefficient Matthews correlation coefficient]'' (a metric used in machine learning for evaluating binary classifications)<br />
<br />
You can force the model evaluation process to run from the command line:<br />
<br />
$ admin/tool/analytics/cli/evaluate_model.php <br />
<br />
<br />
=== Log ===<br />
View previous evaluation logs, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curve graphs, the tensorboard log dir or the model's Matthews correlation coefficient. The information available will depend on the machine learning backend in use.<br />
<br />
=== Edit ===<br />
You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.<br />
<br />
=== Enable / Disable ===<br />
The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.<br />
<br />
=== Export ===<br />
Export your site training data to share it with your partner institutions or to use it on a new site. The Export action for models allows you to generate a csv file containing model data about indicators and weights, without exposing any of your site-specific data. We will be asking for submissions of these model files to help evaluate the value of models on different kinds of sites. Please see the [https://moodle.org/project_inspire Learning Analytics community] for more information.<br />
<br />
=== Invalid site elements ===<br />
Reports on what elements in your site can not be analysed by this model<br />
=== Clear predictions ===<br />
Clears all the model predictions and training data<br />
<br />
[[File:model-evaluation.jpeg]]<br />
<br />
<br />
=== Core models ===<br />
<br />
==== Students at risk of dropping out ====<br />
<br />
[[Students at risk of dropping out|This model]] predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. In this model, the definition of "dropping out" is "no student activity in the final quarter of the course." The prediction model uses the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry] model of student engagement, consisting of three parts:<br />
<br />
* [[Students at risk of dropping out#Cognitive depth|Cognitive presence]]<br />
* [[Students at risk of dropping out#Social Breadth|Social presence]]<br />
* [[Students at risk of dropping out#Potential indicator levels for selected activity modules|Teacher presence]]<br />
<br />
This prediction model is able to analyse and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success in exact duplicates of courses offered in the past. However, there are some limitations:<br />
<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the [[Students at risk of dropping out#Indicators|indicator]] set (see below). Courses which do not include several core Moodle activities per “time slice” (depending on the time splitting method) will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script:<br />
<br />
$ admin/tool/analytics/cli/guess_course_start_and_end.php <br />
<br />
This script attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
<br />
==== No teaching ====<br />
<br />
This model's insights will inform site managers of which courses with an upcoming start date will not have teaching activity. This is a simple model and it does not use machine learning backend to return predictions. It bases the predictions on assumptions, e.g. there is no teaching if there are no students.<br />
<br />
== Predictions and Insights ==<br />
<br />
Models will start generating predictions at different points in time, depending on the site prediction models and the site courses start and end dates. <br />
<br />
Each model defines which predictions will generate [[dev:Analytics API#Insights|insights]] and which predictions will be ignored. For example, the ''[[Students at risk of dropping out]]'' prediction model does not generate an insight if a student is predicted as "not at risk," since the primary interest is which students are at risk of dropping out of courses, not which students are not at risk.<br />
<br />
[[File:prediction-model-insights.jpeg]]<br />
<br />
Users can specify how they wish to receive insights notifications, or turn them off, via their User menu > Preferences > Notification preferences.<br />
<br />
=== Actions===<br />
<br />
Each insight can have one or more actions defined. Actions provide a way to act on the insight as it is read. These actions may include a way to send a message to another user, a link to a report providing information about the sample the prediction has been generated for (e.g. a report for an existing student), or a way to view the details of the model prediction.<br />
<br />
[[Image:08_insights.png|frame|center|Insights and Actions]]<br />
<br />
Insights can also offer two important general actions that are applicable to all insights. First, the user can acknowledge the insight. This removes that particular prediction from the view of the user, e.g. a notification about a particular student at risk is removed from the display.<br />
<br />
The second general action is to mark the insight as "Not useful." This also removes the insight associated with this calculation from the display, but the model is adjusted to make this prediction less likely in the future.<br />
<br />
==Capabilities==<br />
<br />
There are two analytics capabilities:<br />
<br />
* [[Capabilities/moodle/analytics:managemodels|Manage models]] - allowed for the default role of manager only<br />
* [[Capabilities/moodle/analytics:listinsights|List insights]] - allowed for the default roles of manager, teacher and non-editing teacher<br />
<br />
<br />
==Frequently Asked Questions==<br />
[[Moodle Learning Analytics FAQ]]<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Analítica]]<br />
[[de:Analytics]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics&diff=132958Analytics2019-01-14T17:37:28Z<p>Dmonllao: /* Settings */</p>
<hr />
<div>{{Tracking progress}}<br />
==Overview==<br />
<br />
{{MediaPlayer | url = https://youtu.be/MS1IqKsrXAI | desc = Analytics overview}}<br />
<br />
Beginning in version 3.4, Moodle core now implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.<br />
<br />
In Moodle 3.4, this system ships with two built-in models:<br />
<br />
* [[Students at risk of dropping out]]<br />
* No teaching activity<br />
<br />
The system can be easily extended with new custom models, based on reusable targets, indicators, and other components. For more information, see the [[dev:Analytics API| Analytics API]] developer documentation.<br />
<br />
===Features===<br />
* Two built-in prediction models: "[[Students at risk of dropping out]]" and "No Teaching".<br />
* A set of student engagement indicators based on the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry].<br />
* Built-in tools to evaluate models against your site's data<br />
* Proactive notifications for instructors using Events<br />
* Instructors can easily send messages to students identified by the model, or jump to the Outline report for that student for more detail about student activity<br />
* An [[dev:Analytics API| API]] to build indicators and prediction models for third-party Moodle plugins<br />
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends<br />
<br />
'''Note: PHP 7.x is required.'''<br />
<br />
===Limitations===<br />
This release of Moodle Learning Analytics has the following limitations:<br />
<br />
* Models included in this release must be "trained" on a site with previous completed courses, ideally using the Moodle course completion feature. The current models cannot make predictions on a site until this is done.<br />
* The prediction model included with this version requires that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle.<br />
* Models and predictions are only visible to teachers and administrators at present.<br />
<br />
We are continuing to enhance Moodle Learning Analytics, and expanded capabilities will be released going forward. To help contribute to our progress, please join the conversation at the [https://moodle.org/project_inspire Moodle Learning Analytics Community]. In particular, we still need data sets from a wide variety of Moodle-using institutions in order to be able to ship a working prediction model that does not depend on local site data before it can be used.<br />
<br />
== Settings ==<br />
<br />
You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
=== Predictions processor ===<br />
<br />
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python one is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package.<br />
<br />
pip install moodlemlbackend<br />
<br />
=== Time splitting methods ===<br />
<br />
[[dev:Analytics API#Time splitting methods|Time splitting methods]] allow insights generated from one course to be used on another course, even if the two courses are not exactly the same length.<br />
<br />
[[Image:06_timesplitting.png|frame|center|Time splitting methods]]<br />
<br />
Each time splitting method divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the time splitting methods you are interested in using; the evaluation process will iterate through all enabled time-spitting methods, so the more time-splitting methods enabled, the slower the evaluation process will be.<br />
<br />
"Single range" indicates that predictions will be made once, but will take into account a range of time, e.g. one prediction at the end of a course. The prediction is made at the end of the range.<br />
<br />
"No splitting" indicates that the model generates an insight based on a snapshot of data at a given moment, e.g. the "no teaching" model looks to see if there are currently any teachers or students assigned to a course at a defined point before the start of the term, and issues one insight warning the site administrator that no teaching is likely to occur in that empty course.<br />
<br />
"Accumulative" methods differ in how much data is included in the prediction. Both "quarterly" and "quarterly accumulative" predictions are made at the end of each quarter of a time span (e.g. a course), but in "quarterly," only the information from the most recent quarter is included in the prediction, whereas in "quarterly accumulative" all information up to the present is included in the prediction.<br />
<br />
Single range and no splitting methods do not have time constraints. They run during the next scheduled task execution, although models apply different restrictions (e.g. require that a course is finished to use it for training or some data in the course and students to use it to get predictions...). 'Single range' and 'No splitting' are not appropriate for students at risk of dropping out of courses. They are intended to be used in models like 'No teaching' or 'Spammer user' where you just want one prediction and done. To explain this with an example: 'No teaching' model uses 'Single range' time splitting method; the target class (the main PHP class of a model) only accepts courses that will start during the next week. Once we provide 'No teaching' insights for a course we don't provide them anymore. <br />
<br />
The difference between 'Single range' and 'No splitting' is that models analysed using 'Single range' will be limited to the analysable elements (the course in students at risk model) start and end dates, while 'No splitting' do not have any time contraints and all data available in the system is used to calculate the indicators.<br />
<br />
'''Note''': Although the examples above refer to courses, time splitting methods can be used on any analysable element. For example, enrolments can have start and end dates, so a time splitting method could be applied to generate predictions about aspects of an enrollment. For analysable elements with no start and end dates, different time splitting methods would be needed. For example, a "weekly" time splitting method could be applied to a model intended to predict whether a user is likely to log in to the system in the future, on the basis of activity in the previous week.<br />
<br />
===Models output directory===<br />
<br />
[[Image:03_models_output_directory.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server.<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server.<br />
<br />
===Scheduled tasks===<br />
<br />
Most analytics API processes are executed through [[Scheduled_tasks|scheduled tasks]]. These processes usually read the activity log table and can require some time to finish. You can find ''Train models'' and ''Predict models'' scheduled tasks listed in ''Administration > Site administration > Server > Scheduled tasks.'' It is recommended to edit the tasks schedule so they run nightly.<br />
<br />
== Model management ==<br />
<br />
Moodle can support multiple prediction models at once, even within the same course. This can be used for A/B testing to compare the performance and accuracy of multiple models. <br />
<br />
Moodle core ships with two prediction models, ''[[Students at risk of dropping out]]'' and ''No teaching.'' Additional prediction models can be created by using the [[dev:Analytics API| Analytics API]]. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
You can manage your system models from ''Site Administration > Analytics > Analytics models''.<br />
<br />
[[File:prediction-models-list.jpeg]]<br />
<br />
These are some of the actions you can perform on a model:<br />
<br />
=== View Insights ===<br />
Once you have trained a machine learning algorithm with the data available on the system, you will see insights (predictions) here for each "analysable." In the included model "[[Students at risk of dropping out]], insights may be selected per course. ''Predictions are not limited to ongoing courses-- this depends on the model.''<br />
<br />
=== Evaluate ===<br />
(disabled by default) Evaluate the prediction model by getting all the training data available on the site, calculating all the indicators and the target and passing the resulting dataset to machine learning backends. This process will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process uses all information available on the site, even if it is very old. Because of this, the accuracy returned by the evaluation process may be lower than the real model accuracy as indicators are more reliably calculated immediately after training data is available because the site state changes over time. The metric used to describe accuracy is the ''[https://en.wikipedia.org/wiki/Matthews_correlation_coefficient Matthews correlation coefficient]'' (a metric used in machine learning for evaluating binary classifications)<br />
<br />
You can force the model evaluation process to run from the command line:<br />
<br />
$ admin/tool/analytics/cli/evaluate_model.php <br />
<br />
<br />
=== Log ===<br />
View previous evaluation logs, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curve graphs, the tensorboard log dir or the model's Matthews correlation coefficient. The information available will depend on the machine learning backend in use.<br />
<br />
=== Edit ===<br />
You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.<br />
<br />
=== Enable / Disable ===<br />
The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.<br />
<br />
=== Export ===<br />
Export your site training data to share it with your partner institutions or to use it on a new site. The Export action for models allows you to generate a csv file containing model data about indicators and weights, without exposing any of your site-specific data. We will be asking for submissions of these model files to help evaluate the value of models on different kinds of sites. Please see the [https://moodle.org/project_inspire Learning Analytics community] for more information.<br />
<br />
=== Invalid site elements ===<br />
Reports on what elements in your site can not be analysed by this model<br />
=== Clear predictions ===<br />
Clears all the model predictions and training data<br />
<br />
[[File:model-evaluation.jpeg]]<br />
<br />
<br />
=== Core models ===<br />
<br />
==== Students at risk of dropping out ====<br />
<br />
[[Students at risk of dropping out|This model]] predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. In this model, the definition of "dropping out" is "no student activity in the final quarter of the course." The prediction model uses the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry] model of student engagement, consisting of three parts:<br />
<br />
* [[Students at risk of dropping out#Cognitive depth|Cognitive presence]]<br />
* [[Students at risk of dropping out#Social Breadth|Social presence]]<br />
* [[Students at risk of dropping out#Potential indicator levels for selected activity modules|Teacher presence]]<br />
<br />
This prediction model is able to analyse and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success in exact duplicates of courses offered in the past. However, there are some limitations:<br />
<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the [[Students at risk of dropping out#Indicators|indicator]] set (see below). Courses which do not include several core Moodle activities per “time slice” (depending on the time splitting method) will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script:<br />
<br />
$ admin/tool/analytics/cli/guess_course_start_and_end.php <br />
<br />
This script attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
<br />
==== No teaching ====<br />
<br />
This model's insights will inform site managers of which courses with an upcoming start date will not have teaching activity. This is a simple model and it does not use machine learning backend to return predictions. It bases the predictions on assumptions, e.g. there is no teaching if there are no students.<br />
<br />
== Predictions and Insights ==<br />
<br />
Models will start generating predictions at different points in time, depending on the site prediction models and the site courses start and end dates. <br />
<br />
Each model defines which predictions will generate [[dev:Analytics API#Insights|insights]] and which predictions will be ignored. For example, the ''[[Students at risk of dropping out]]'' prediction model does not generate an insight if a student is predicted as "not at risk," since the primary interest is which students are at risk of dropping out of courses, not which students are not at risk.<br />
<br />
[[File:prediction-model-insights.jpeg]]<br />
<br />
Users can specify how they wish to receive insights notifications, or turn them off, via their User menu > Preferences > Notification preferences.<br />
<br />
=== Actions===<br />
<br />
Each insight can have one or more actions defined. Actions provide a way to act on the insight as it is read. These actions may include a way to send a message to another user, a link to a report providing information about the sample the prediction has been generated for (e.g. a report for an existing student), or a way to view the details of the model prediction.<br />
<br />
[[Image:08_insights.png|frame|center|Insights and Actions]]<br />
<br />
Insights can also offer two important general actions that are applicable to all insights. First, the user can acknowledge the insight. This removes that particular prediction from the view of the user, e.g. a notification about a particular student at risk is removed from the display.<br />
<br />
The second general action is to mark the insight as "Not useful." This also removes the insight associated with this calculation from the display, but the model is adjusted to make this prediction less likely in the future.<br />
<br />
==Capabilities==<br />
<br />
There are two analytics capabilities:<br />
<br />
* [[Capabilities/moodle/analytics:managemodels|Manage models]] - allowed for the default role of manager only<br />
* [[Capabilities/moodle/analytics:listinsights|List insights]] - allowed for the default roles of manager, teacher and non-editing teacher<br />
<br />
<br />
==Frequently Asked Questions==<br />
[[Moodle Learning Analytics FAQ]]<br />
<br />
[[Category:Analytics]]<br />
<br />
[[es:Analítica]]<br />
[[de:Analytics]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=131587Global search2018-08-01T12:15:26Z<p>Dmonllao: /* The Solr server */</p>
<hr />
<div>{{More features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator in Advanced features and then, from the Manage Global search page, the search engine must be selected.<br />
<br />
When Global search is enabled, search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==What can I search for?==<br />
You can search for your courses, activities information and some activities contents like forum posts, book chapters, glossary entries or collaborative wikis pages. <br />
<br />
HTML block content may be searched and, if the administrator sets the setting "Searchable courses" to ''Search within all courses the user can access'' then courses which are visible to users but which they are not enrolled in may also be searched. (These courses are courses with guest access or if the user has the capability to view all courses.)<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
===Solr-specific features===<br />
''These features are only available if you use '''Solr''' as your search engine.''<br />
<br />
====Search query features====<br />
You can improve your search using any of the following search query features:<br />
<br />
* Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title<br />
* Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords<br />
* Wildcard characters ('*' or '?' ) to represent characters in the search query<br />
* Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)<br />
* Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.<br />
<br />
====File contents indexing====<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts.<br />
<br />
====Extra filters====<br />
'''New in 3.5''' : New fields now facilitate filtering by user ID and searching by group.<br />
<br />
==How is it set up?==<br />
<br />
Note: Global search needs a search engine. It's recommended you set everything up first, index the site contents ''and only then'' enable Global search.<br />
<br />
Moodle core includes two search engines: Simple Global search and Solr. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own.<br />
<br />
===Setting up Simple search===<br />
<br />
{{New features}}<br />
Simple Global search provides global search features without the need to install an external search engine.<br />
<br />
{{MediaPlayer | url = https://youtu.be/4JJhOd2XAiY | desc = Simple global search}}<br />
<br />
===Setting up Solr===<br />
<br />
* The latest Solr 5 available version is the recommended one.<br />
* Moodle supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released.<br />
* Solr 7 is not currently supported as the field type 'int' has been removed.<br />
<br />
====General Setup====<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# You now need to populate the created Solr index with your site's data. You can do it via the web interface by going to ''Site administration > Reports > Global search info'' or from the CLI by running the <tt>search/cli/indexer.php</tt> script. The CLI script is the recommended option for big sites.<br><code># sudo -u www-run php search/cli/indexer.php --force</code><br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
====File Indexing====<br />
<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the [https://tika.apache.org/ Tika] engine which comes bundled with Solr. To enable this feature:<br />
# In ''Site administration > Plugins > Search > Solr'' enable the checkbox '''Enable file indexing'''<br />
# Set '''Maximum file size to index (kB)''' to some value - the default is '''2097152''' (2GB)<br />
#* Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.<br />
<br />
'''New in 34:''' File indexing now works retroactively -ie, if you set up Global search without file indexing enabled and a later date you enable file indexing, files associated with existing objects will be included.<br />
<br />
====How to install Solr====<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr PECL: Package: solr]. The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
===== Linux (Debian/Ubuntu) =====<br />
With PHP5.x<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
With PHP 7<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php7.0-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
===== Linux (Redhat/Centos 6 & 7) =====<br />
<br />
====== Using built in php5 packages ======<br />
yum install php-pecl-solr2<br />
service httpd restart<br />
<br />
====== Using 3rd party php7 packages (webtactic) ======<br />
yum install libxml2-devel pcre-devel libcurl-devel php70w-devel php70w-pear<br />
pecl install solr<br />
echo "extension=solr.so" > /etc/php.d/solr.ini<br />
service httpd restart<br />
<br />
===== OSX using macports =====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
===== OSX using homebrew =====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
===== Windows =====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
===== The Solr server =====<br />
<br />
Note that for medium/large sites you may need to increase maxBooleanClauses setting. In [https://tracker.moodle.org/browse/MDL-54992 MDL-54992] we are working on an alternative way to query the server. <br />
<br />
Please note that there is a security issue that affects some Solr versions. The patched versions are 5.5.5, 6.6.2, 7.1, 7.2 and all new versions are patched. If your Solr server version is lower but you can't upgrade to one of the patched versions it is recommended to disable the XML Query Parser.<br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.5.5 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it. If wget gives an error, check http://www-us.apache.org/dist/lucene/solr and update SOLRVERSION<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.5.5<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
<br />
======Solr 5/6 schema setup======<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
======Solr 4 schema setup======<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
======Memory and File indexing considerations======<br />
<br />
For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, eg aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the ''-m'' option: <br />
solr start -m 12g<br />
<br />
See the documents for your version of Solr on how to increases memory when Solr is started automatically.<br />
<br />
======Too many boolean clauses error======<br />
<br />
Because of the way Moodle handles permissions for searches, if you have non-admin users with access to a large number of activities (>1000), they may an error similar to the following:<br />
<br />
The query you provided could not be parsed by the search engine: org.apache.solr.search.SyntaxError: Cannot parse <br />
...<br />
too many boolean clauses<br />
<br />
To correct this, you need to increase the ''maxBooleanClauses'' setting in your Solr core. The setting is located in ''corename/conf/solrconfig.xml''.<br />
<br />
This important note is included in the config file:<br />
** WARNING **<br />
<br />
This option actually modifies a global Lucene property that<br />
will affect all SolrCores. If multiple solrconfig.xml files<br />
disagree on this property, the value at any given moment will<br />
be based on the last SolrCore to be initialized.<br />
<br />
This means that for consistent behavior you should update this value for all cores in the Solr server.<br />
<br />
=====Commannd for installing Solr Server on Centos 7=====<br />
First of all install java (openjdk)<br />
yum install java-1.8.0-openjdk<br />
Add user for solr<br />
adduser solr<br />
Download solr server, find latest here : http://lucene.apache.org/solr/mirrors-solr-latest-redir.html, from the list there download latest solr-[version].tgz<br />
<br />
Here is the list of needed commands (note that latest solr is suggested, change command to appropriate version) <br />
cd /opt<br />
wget http://www-eu.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz<br />
tar zxvf solr-6.6.0.tgz<br />
cp solr-6.6.0/bin/install_solr_service.sh .<br />
rm -rf solr-6.6.0<br />
./install_solr_service.sh solr-6.6.0.tgz<br />
chkconfig solr on<br />
Create the index<br />
su - solr -c "/opt/solr/bin/solr create_core -c moodle"<br />
Then go to your Moodle and set index name as created (moodle)<br />
<br />
==See also==<br />
<br />
Forum discussions:<br />
* [https://moodle.org/mod/forum/discuss.php?d=334938 Global search doesn't index plugins?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=339819 Global Search on Moodle 3.1]<br />
<br />
<br />
A quick introduction to Moodle global search, followed by a description of the new features and fixes in Moodle 3.4 (and some of those in Moodle 3.5).<br />
*[https://www.slideshare.net/sammarshall_ou/moodle-global-search-nows-the-time-to-jump-in Presentation by Sam Marshall from the Open University]<br />
<br />
[[es:Búsqueda global]]<br />
[[de:Globale Suche]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=131100Global search2018-05-23T15:04:41Z<p>Dmonllao: /* How does it work? */</p>
<hr />
<div>{{More features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator in Advanced features and then, from the Manage Global search page, the search engine must be selected.<br />
<br />
When Global search is enabled, search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==What can I search for?==<br />
You can search for your courses, activities information and some activities contents like forum posts, book chapters, glossary entries or collaborative wikis pages. <br />
<br />
HTML block content may be searched and, if the administrator sets the setting "Searchable courses" to ''Search within all courses the user can access'' then courses which are visible to users but which they are not enrolled in may also be searched. (These courses are courses with guest access or if the user has the capability to view all courses.)<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
===Solr-specific features===<br />
''These features are only available if you use '''Solr''' as your search engine.''<br />
<br />
====Search query features====<br />
You can improve your search using any of the following search query features:<br />
<br />
* Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title<br />
* Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords<br />
* Wildcard characters ('*' or '?' ) to represent characters in the search query<br />
* Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)<br />
* Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.<br />
<br />
====File contents indexing====<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts.<br />
<br />
====Extra filters====<br />
'''New in 3.5''' : New fields now facilitate filtering by user ID and searching by group.<br />
<br />
==How is it set up?==<br />
<br />
Note: Global search needs a search engine. It's recommended you set everything up first, index the site contents ''and only then'' enable Global search.<br />
<br />
Moodle core includes two search engines: Simple Global search and Solr. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own.<br />
<br />
===Setting up Simple search===<br />
<br />
{{New features}}<br />
Simple Global search provides global search features without the need to install an external search engine.<br />
<br />
{{MediaPlayer | url = https://youtu.be/4JJhOd2XAiY | desc = Simple global search}}<br />
<br />
===Setting up Solr===<br />
<br />
* The latest Solr 5 available version is the recommended one.<br />
* Moodle supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released.<br />
* Solr 7 is not currently supported as the field type 'int' has been removed.<br />
<br />
====General Setup====<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# You now need to populate the created Solr index with your site's data. You can do it via the web interface by going to ''Site administration > Reports > Global search info'' or from the CLI by running the <tt>search/cli/indexer.php</tt> script. The CLI script is the recommended option for big sites.<br><code># sudo -u www-run php search/cli/indexer.php --force</code><br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
====File Indexing====<br />
<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the [https://tika.apache.org/ Tika] engine which comes bundled with Solr. To enable this feature:<br />
# In ''Site administration > Plugins > Search > Solr'' enable the checkbox '''Enable file indexing'''<br />
# Set '''Maximum file size to index (kB)''' to some value - the default is '''2097152''' (2GB)<br />
#* Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.<br />
<br />
'''New in 34:''' File indexing now works retroactively -ie, if you set up Global search without file indexing enabled and a later date you enable file indexing, files associated with existing objects will be included.<br />
<br />
====How to install Solr====<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr PECL: Package: solr]. The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
===== Linux (Debian/Ubuntu) =====<br />
With PHP5.x<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
With PHP 7<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php7.0-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
===== Linux (Redhat/Centos 6 & 7) =====<br />
<br />
====== Using built in php5 packages ======<br />
yum install php-pecl-solr2<br />
service httpd restart<br />
<br />
====== Using 3rd party php7 packages (webtactic) ======<br />
yum install libxml2-devel pcre-devel libcurl-devel php70w-devel php70w-pear<br />
pecl install solr<br />
echo "extension=solr.so" > /etc/php.d/solr.ini<br />
service httpd restart<br />
<br />
===== OSX using macports =====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
===== OSX using homebrew =====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
===== Windows =====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
===== The Solr server =====<br />
<br />
Note that for medium/large sites you may need to increase maxBooleanClauses setting. In [https://tracker.moodle.org/browse/MDL-54992 MDL-54992] we are working on an alternative way to query the server. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it. If wget gives an error, check http://www-us.apache.org/dist/lucene/solr and update SOLRVERSION<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.5.5<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
<br />
======Solr 5/6 schema setup======<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
======Solr 4 schema setup======<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
======Memory and File indexing considerations======<br />
<br />
For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, eg aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the ''-m'' option: <br />
solr start -m 12g<br />
<br />
See the documents for your version of Solr on how to increases memory when Solr is started automatically.<br />
<br />
======Too many boolean clauses error======<br />
<br />
Because of the way Moodle handles permissions for searches, if you have non-admin users with access to a large number of activities (>1000), they may an error similar to the following:<br />
<br />
The query you provided could not be parsed by the search engine: org.apache.solr.search.SyntaxError: Cannot parse <br />
...<br />
too many boolean clauses<br />
<br />
To correct this, you need to increase the ''maxBooleanClauses'' setting in your Solr core. The setting is located in ''corename/conf/solrconfig.xml''.<br />
<br />
This important note is included in the config file:<br />
** WARNING **<br />
<br />
This option actually modifies a global Lucene property that<br />
will affect all SolrCores. If multiple solrconfig.xml files<br />
disagree on this property, the value at any given moment will<br />
be based on the last SolrCore to be initialized.<br />
<br />
This means that for consistent behavior you should update this value for all cores in the Solr server.<br />
<br />
=====Commannd for installing Solr Server on Centos 7=====<br />
First of all install java (openjdk)<br />
yum install java-1.8.0-openjdk<br />
Add user for solr<br />
adduser solr<br />
Download solr server, find latest here : http://lucene.apache.org/solr/mirrors-solr-latest-redir.html, from the list there download latest solr-[version].tgz<br />
<br />
Here is the list of needed commands (note that latest solr is suggested, change command to appropriate version) <br />
cd /opt<br />
wget http://www-eu.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz<br />
tar zxvf solr-6.6.0.tgz<br />
cp solr-6.6.0/bin/install_solr_service.sh .<br />
rm -rf solr-6.6.0<br />
./install_solr_service.sh solr-6.6.0.tgz<br />
chkconfig solr on<br />
Create the index<br />
su - solr -c "/opt/solr/bin/solr create_core -c moodle"<br />
Then go to your Moodle and set index name as created (moodle)<br />
<br />
==See also==<br />
<br />
Forum discussions:<br />
* [https://moodle.org/mod/forum/discuss.php?d=334938 Global search doesn't index plugins?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=339819 Global Search on Moodle 3.1]<br />
<br />
[[es:Búsqueda global]]<br />
[[de:Globale Suche]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=131099Global search2018-05-23T14:57:10Z<p>Dmonllao: /* How is it set up? */</p>
<hr />
<div>{{More features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator in Advanced features and then, from the Manage Global search page, the search engine must be selected.<br />
<br />
When Global search is enabled, search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==What can I search for?==<br />
You can search for your courses, activities information and some activities contents like forum posts, book chapters, glossary entries or collaborative wikis pages. <br />
<br />
HTML block content may be searched and, if the administrator sets the setting "Searchable courses" to ''Search within all courses the user can access'' then courses which are visible to users but which they are not enrolled in may also be searched. (These courses are courses with guest access or if the user has the capability to view all courses.)<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
===Solr search query features===<br />
<br />
''These search features are only available if you use '''Solr''' as your search engine.''<br />
<br />
You can improve your search using any of the following search query features:<br />
<br />
* Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title<br />
* Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords<br />
* Wildcard characters ('*' or '?' ) to represent characters in the search query<br />
* Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)<br />
* Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.<br />
<br />
'''New in 3.5''' : New fields now facilitate filtering by user ID and searching by group.<br />
<br />
Note: The above features are only available when using the Solr search engine.<br />
<br />
==How is it set up?==<br />
<br />
Note: Global search needs a search engine. It's recommended you set everything up first, index the site contents ''and only then'' enable Global search.<br />
<br />
Moodle core includes two search engines: Simple Global search and Solr. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own.<br />
<br />
===Setting up Simple search===<br />
<br />
{{New features}}<br />
Simple Global search provides global search features without the need to install an external search engine.<br />
<br />
{{MediaPlayer | url = https://youtu.be/4JJhOd2XAiY | desc = Simple global search}}<br />
<br />
===Setting up Solr===<br />
<br />
* The latest Solr 5 available version is the recommended one.<br />
* Moodle supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released.<br />
* Solr 7 is not currently supported as the field type 'int' has been removed.<br />
<br />
====General Setup====<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# You now need to populate the created Solr index with your site's data. You can do it via the web interface by going to ''Site administration > Reports > Global search info'' or from the CLI by running the <tt>search/cli/indexer.php</tt> script. The CLI script is the recommended option for big sites.<br><code># sudo -u www-run php search/cli/indexer.php --force</code><br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
====File Indexing====<br />
<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the [https://tika.apache.org/ Tika] engine which comes bundled with Solr. To enable this feature:<br />
# In ''Site administration > Plugins > Search > Solr'' enable the checkbox '''Enable file indexing'''<br />
# Set '''Maximum file size to index (kB)''' to some value - the default is '''2097152''' (2GB)<br />
#* Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.<br />
<br />
'''New in 34:''' File indexing now works retroactively -ie, if you set up Global search without file indexing enabled and a later date you enable file indexing, files associated with existing objects will be included.<br />
<br />
====How to install Solr====<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr PECL: Package: solr]. The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
===== Linux (Debian/Ubuntu) =====<br />
With PHP5.x<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
With PHP 7<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php7.0-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
===== Linux (Redhat/Centos 6 & 7) =====<br />
<br />
====== Using built in php5 packages ======<br />
yum install php-pecl-solr2<br />
service httpd restart<br />
<br />
====== Using 3rd party php7 packages (webtactic) ======<br />
yum install libxml2-devel pcre-devel libcurl-devel php70w-devel php70w-pear<br />
pecl install solr<br />
echo "extension=solr.so" > /etc/php.d/solr.ini<br />
service httpd restart<br />
<br />
===== OSX using macports =====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
===== OSX using homebrew =====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
===== Windows =====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
===== The Solr server =====<br />
<br />
Note that for medium/large sites you may need to increase maxBooleanClauses setting. In [https://tracker.moodle.org/browse/MDL-54992 MDL-54992] we are working on an alternative way to query the server. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it. If wget gives an error, check http://www-us.apache.org/dist/lucene/solr and update SOLRVERSION<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.5.5<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
<br />
======Solr 5/6 schema setup======<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
======Solr 4 schema setup======<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
======Memory and File indexing considerations======<br />
<br />
For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, eg aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the ''-m'' option: <br />
solr start -m 12g<br />
<br />
See the documents for your version of Solr on how to increases memory when Solr is started automatically.<br />
<br />
======Too many boolean clauses error======<br />
<br />
Because of the way Moodle handles permissions for searches, if you have non-admin users with access to a large number of activities (>1000), they may an error similar to the following:<br />
<br />
The query you provided could not be parsed by the search engine: org.apache.solr.search.SyntaxError: Cannot parse <br />
...<br />
too many boolean clauses<br />
<br />
To correct this, you need to increase the ''maxBooleanClauses'' setting in your Solr core. The setting is located in ''corename/conf/solrconfig.xml''.<br />
<br />
This important note is included in the config file:<br />
** WARNING **<br />
<br />
This option actually modifies a global Lucene property that<br />
will affect all SolrCores. If multiple solrconfig.xml files<br />
disagree on this property, the value at any given moment will<br />
be based on the last SolrCore to be initialized.<br />
<br />
This means that for consistent behavior you should update this value for all cores in the Solr server.<br />
<br />
=====Commannd for installing Solr Server on Centos 7=====<br />
First of all install java (openjdk)<br />
yum install java-1.8.0-openjdk<br />
Add user for solr<br />
adduser solr<br />
Download solr server, find latest here : http://lucene.apache.org/solr/mirrors-solr-latest-redir.html, from the list there download latest solr-[version].tgz<br />
<br />
Here is the list of needed commands (note that latest solr is suggested, change command to appropriate version) <br />
cd /opt<br />
wget http://www-eu.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz<br />
tar zxvf solr-6.6.0.tgz<br />
cp solr-6.6.0/bin/install_solr_service.sh .<br />
rm -rf solr-6.6.0<br />
./install_solr_service.sh solr-6.6.0.tgz<br />
chkconfig solr on<br />
Create the index<br />
su - solr -c "/opt/solr/bin/solr create_core -c moodle"<br />
Then go to your Moodle and set index name as created (moodle)<br />
<br />
==See also==<br />
<br />
Forum discussions:<br />
* [https://moodle.org/mod/forum/discuss.php?d=334938 Global search doesn't index plugins?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=339819 Global Search on Moodle 3.1]<br />
<br />
[[es:Búsqueda global]]<br />
[[de:Globale Suche]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=131098Global search2018-05-23T14:46:19Z<p>Dmonllao: /* What can I search for? */</p>
<hr />
<div>{{More features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator in Advanced features and then, from the Manage Global search page, the search engine must be selected.<br />
<br />
When Global search is enabled, search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==What can I search for?==<br />
You can search for your courses, activities information and some activities contents like forum posts, book chapters, glossary entries or collaborative wikis pages. <br />
<br />
HTML block content may be searched and, if the administrator sets the setting "Searchable courses" to ''Search within all courses the user can access'' then courses which are visible to users but which they are not enrolled in may also be searched. (These courses are courses with guest access or if the user has the capability to view all courses.)<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
===Solr search query features===<br />
<br />
''These search features are only available if you use '''Solr''' as your search engine.''<br />
<br />
You can improve your search using any of the following search query features:<br />
<br />
* Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title<br />
* Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords<br />
* Wildcard characters ('*' or '?' ) to represent characters in the search query<br />
* Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)<br />
* Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.<br />
<br />
'''New in 3.5''' : New fields now facilitate filtering by user ID and searching by group.<br />
<br />
Note: The above features are only available when using the Solr search engine.<br />
<br />
==How is it set up?==<br />
<br />
Note: Global search needs a search engine. It's recommended you set everything up first, index the site contents ''and only then'' enable Global search.<br />
<br />
Moodle core includes two search engines: Simple Global search and Solr. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own.<br />
<br />
===Setting up Simple search===<br />
<br />
{{New features}}<br />
Simple Global search provides global search features without the need to install an external search engine.<br />
<br />
{{MediaPlayer | url = https://youtu.be/4JJhOd2XAiY | desc = Simple global search}}<br />
<br />
===Setting up Solr===<br />
<br />
* The latest Solr 5 available version is the recommended one.<br />
* Moodle supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released.<br />
* Solr 7 is not currently supported as the field type 'int' has been removed.<br />
<br />
====General Setup====<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# You now need to populate the created Solr index with your site's data. You can do it via the web interface by going to ''Site administration > Reports > Global search info'' or from the CLI by running the <tt>search/cli/indexer.php</tt> script. The CLI script is the recommended option for big sites.<br><code># sudo -u www-run php search/cli/indexer.php --force</code><br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
====File Indexing====<br />
<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the [https://tika.apache.org/ Tika] engine which comes bundled with Solr. To enable this feature:<br />
# In ''Site administration > Plugins > Search > Solr'' enable the checkbox '''Enable file indexing'''<br />
# Set '''Maximum file size to index (kB)''' to some value - the default is '''2097152''' (2GB)<br />
#* Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.<br />
<br />
'''New in 34:''' File indexing now works retroactively -ie, if you set up Global search without file indexing enabled and a later date you enable file indexing, files associated with existing objects will be included.<br />
<br />
===How to install Solr===<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr PECL: Package: solr]. The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
==== Linux (Debian/Ubuntu) ====<br />
With PHP5.x<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
With PHP 7<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php7.0-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
==== Linux (Redhat/Centos 6 & 7) ====<br />
<br />
===== Using built in php5 packages =====<br />
yum install php-pecl-solr2<br />
service httpd restart<br />
<br />
===== Using 3rd party php7 packages (webtactic) =====<br />
yum install libxml2-devel pcre-devel libcurl-devel php70w-devel php70w-pear<br />
pecl install solr<br />
echo "extension=solr.so" > /etc/php.d/solr.ini<br />
service httpd restart<br />
<br />
==== OSX using macports ====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
==== OSX using homebrew ====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
==== Windows ====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
==== The Solr server ====<br />
<br />
Note that for medium/large sites you may need to increase maxBooleanClauses setting. In [https://tracker.moodle.org/browse/MDL-54992 MDL-54992] we are working on an alternative way to query the server. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it. If wget gives an error, check http://www-us.apache.org/dist/lucene/solr and update SOLRVERSION<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.5.5<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
<br />
=====Solr 5/6 schema setup=====<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
=====Solr 4 schema setup=====<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
=====Memory and File indexing considerations=====<br />
<br />
For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, eg aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the ''-m'' option: <br />
solr start -m 12g<br />
<br />
See the documents for your version of Solr on how to increases memory when Solr is started automatically.<br />
<br />
=====Too many boolean clauses error=====<br />
<br />
Because of the way Moodle handles permissions for searches, if you have non-admin users with access to a large number of activities (>1000), they may an error similar to the following:<br />
<br />
The query you provided could not be parsed by the search engine: org.apache.solr.search.SyntaxError: Cannot parse <br />
...<br />
too many boolean clauses<br />
<br />
To correct this, you need to increase the ''maxBooleanClauses'' setting in your Solr core. The setting is located in ''corename/conf/solrconfig.xml''.<br />
<br />
This important note is included in the config file:<br />
** WARNING **<br />
<br />
This option actually modifies a global Lucene property that<br />
will affect all SolrCores. If multiple solrconfig.xml files<br />
disagree on this property, the value at any given moment will<br />
be based on the last SolrCore to be initialized.<br />
<br />
This means that for consistent behavior you should update this value for all cores in the Solr server.<br />
<br />
====Commannd for installing Solr Server on Centos 7====<br />
First of all install java (openjdk)<br />
yum install java-1.8.0-openjdk<br />
Add user for solr<br />
adduser solr<br />
Download solr server, find latest here : http://lucene.apache.org/solr/mirrors-solr-latest-redir.html, from the list there download latest solr-[version].tgz<br />
<br />
Here is the list of needed commands (note that latest solr is suggested, change command to appropriate version) <br />
cd /opt<br />
wget http://www-eu.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz<br />
tar zxvf solr-6.6.0.tgz<br />
cp solr-6.6.0/bin/install_solr_service.sh .<br />
rm -rf solr-6.6.0<br />
./install_solr_service.sh solr-6.6.0.tgz<br />
chkconfig solr on<br />
Create the index<br />
su - solr -c "/opt/solr/bin/solr create_core -c moodle"<br />
Then go to your Moodle and set index name as created (moodle)<br />
<br />
==See also==<br />
<br />
Forum discussions:<br />
* [https://moodle.org/mod/forum/discuss.php?d=334938 Global search doesn't index plugins?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=339819 Global Search on Moodle 3.1]<br />
<br />
[[es:Búsqueda global]]<br />
[[de:Globale Suche]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=131064Global search2018-05-17T11:24:36Z<p>Dmonllao: </p>
<hr />
<div>{{More features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator in Advanced features and then, from the Manage Global search page, the search engine must be selected.<br />
<br />
When Global search is enabled, search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==What can I search for?==<br />
You can search for your courses, activities information and some activities contents like forum posts, book chapters, glossary entries or collaborative wikis pages. <br />
<br />
HTML block content may be searched and, if the administrator enables the setting "Search all courses" then courses which are visible to users but which they are not enrolled in may also be searched. (These courses are courses with guest access or if the user has the capability to view all courses.)<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
===Solr search query features===<br />
<br />
''These search features are only available if you use '''Solr''' as your search engine.''<br />
<br />
You can improve your search using any of the following search query features:<br />
<br />
* Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title<br />
* Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords<br />
* Wildcard characters ('*' or '?' ) to represent characters in the search query<br />
* Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)<br />
* Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.<br />
<br />
'''New in 3.5''' : New fields now facilitate filtering by user ID and searching by group.<br />
<br />
Note: The above features are only available when using the Solr search engine.<br />
<br />
==How is it set up?==<br />
<br />
Note: Global search needs a search engine. It's recommended you set everything up first, index the site contents ''and only then'' enable Global search.<br />
<br />
Moodle core includes two search engines: Simple Global search and Solr. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own.<br />
<br />
===Setting up Simple search===<br />
<br />
{{New features}}<br />
Simple Global search provides global search features without the need to install an external search engine.<br />
<br />
{{MediaPlayer | url = https://youtu.be/4JJhOd2XAiY | desc = Simple global search}}<br />
<br />
===Setting up Solr===<br />
<br />
* The latest Solr 5 available version is the recommended one.<br />
* Moodle supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released.<br />
* Solr 7 is not currently supported as the field type 'int' has been removed.<br />
<br />
====General Setup====<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# You now need to populate the created Solr index with your site's data. You can do it via the web interface by going to ''Site administration > Reports > Global search info'' or from the CLI by running the <tt>search/cli/indexer.php</tt> script. The CLI script is the recommended option for big sites.<br><code># sudo -u www-run php search/cli/indexer.php --force</code><br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
====File Indexing====<br />
<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the [https://tika.apache.org/ Tika] engine which comes bundled with Solr. To enable this feature:<br />
# In ''Site administration > Plugins > Search > Solr'' enable the checkbox '''Enable file indexing'''<br />
# Set '''Maximum file size to index (kB)''' to some value - the default is '''2097152''' (2GB)<br />
#* Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.<br />
<br />
'''New in 34:''' File indexing now works retroactively -ie, if you set up Global search without file indexing enabled and a later date you enable file indexing, files associated with existing objects will be included.<br />
<br />
===How to install Solr===<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr PECL: Package: solr]. The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
==== Linux (Debian/Ubuntu) ====<br />
With PHP5.x<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
With PHP 7<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php7.0-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php/7.0/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
==== Linux (Redhat/Centos 6 & 7) ====<br />
<br />
===== Using built in php5 packages =====<br />
yum install php-pecl-solr2<br />
service httpd restart<br />
<br />
===== Using 3rd party php7 packages (webtactic) =====<br />
yum install libxml2-devel pcre-devel libcurl-devel php70w-devel php70w-pear<br />
pecl install solr<br />
echo "extension=solr.so" > /etc/php.d/solr.ini<br />
service httpd restart<br />
<br />
==== OSX using macports ====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
==== OSX using homebrew ====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
==== Windows ====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
==== The Solr server ====<br />
<br />
Note that for medium/large sites you may need to increase maxBooleanClauses setting. In [https://tracker.moodle.org/browse/MDL-54992 MDL-54992] we are working on an alternative way to query the server. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it. If wget gives an error, check http://www-us.apache.org/dist/lucene/solr and update SOLRVERSION<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.4.1<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
<br />
=====Solr 5/6 schema setup=====<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
=====Solr 4 schema setup=====<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
=====Memory and File indexing considerations=====<br />
<br />
For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, eg aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the ''-m'' option: <br />
solr start -m 12g<br />
<br />
See the documents for your version of Solr on how to increases memory when Solr is started automatically.<br />
<br />
=====Too many boolean clauses error=====<br />
<br />
Because of the way Moodle handles permissions for searches, if you have non-admin users with access to a large number of activities (>1000), they may an error similar to the following:<br />
<br />
The query you provided could not be parsed by the search engine: org.apache.solr.search.SyntaxError: Cannot parse <br />
...<br />
too many boolean clauses<br />
<br />
To correct this, you need to increase the ''maxBooleanClauses'' setting in your Solr core. The setting is located in ''corename/conf/solrconfig.xml''.<br />
<br />
This important note is included in the config file:<br />
** WARNING **<br />
<br />
This option actually modifies a global Lucene property that<br />
will affect all SolrCores. If multiple solrconfig.xml files<br />
disagree on this property, the value at any given moment will<br />
be based on the last SolrCore to be initialized.<br />
<br />
This means that for consistent behavior you should update this value for all cores in the Solr server.<br />
<br />
====Commannd for installing Solr Server on Centos 7====<br />
First of all install java (openjdk)<br />
yum install java-1.8.0-openjdk<br />
Add user for solr<br />
adduser solr<br />
Download solr server, find latest here : http://lucene.apache.org/solr/mirrors-solr-latest-redir.html, from the list there download latest solr-[version].tgz<br />
<br />
Here is the list of needed commands (note that latest solr is suggested, change command to appropriate version) <br />
cd /opt<br />
wget http://www-eu.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz<br />
tar zxvf solr-6.6.0.tgz<br />
cp solr-6.6.0/bin/install_solr_service.sh .<br />
rm -rf solr-6.6.0<br />
./install_solr_service.sh solr-6.6.0.tgz<br />
chkconfig solr on<br />
Create the index<br />
su - solr -c "/opt/solr/bin/solr create_core -c moodle"<br />
Then go to your Moodle and set index name as created (moodle)<br />
<br />
==See also==<br />
<br />
Forum discussions:<br />
* [https://moodle.org/mod/forum/discuss.php?d=334938 Global search doesn't index plugins?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=339819 Global Search on Moodle 3.1]<br />
<br />
[[es:Búsqueda global]]<br />
[[de:Globale Suche]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics&diff=129239Analytics2017-11-10T08:27:26Z<p>Dmonllao: /* Predictions processor */</p>
<hr />
<div>{{Tracking progress}}{{New features}}<br />
<br />
==Overview==<br />
This page describes analytics features in Moodle core. Open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success to learners and teachers.<br />
<br />
== Settings ==<br />
<br />
You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
=== Predictions processor ===<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP one is the default. There are no other system requirements to use this processor.<br />
* The Python one is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra stuff: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package.<br />
<br />
pip install moodlemlbackend<br />
<br />
=== Time splitting methods ===<br />
<br />
The time splitting method divides the course duration in parts, the predictions engine will run at the end of these parts. It is recommended that you only enable the time splitting methods you could be interested on using; the site contents analyser will calculate all indicators using each of the enabled time splitting methods. The more enabled time splitting methods the slower the evaluation process will be.<br />
<br />
===Models output directory===<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server.<br />
<br />
== Models management ==<br />
<br />
Moodle can support multiple prediction models at once. Moodle core comes with two prediction models ''Students at risk of dropping out'' and ''No teaching'', you can add your own prediction models using [https://docs.moodle.org/dev/Analytics_API Moodle analytics API]. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
You can manage your system models from ''Site Administration > Analytics > Analytics models''.<br />
<br />
[[File:prediction-models-list.jpeg]]<br />
<br />
These are some of the actions you can perform on a model:<br />
<br />
* '''Get predictions:''' Train machine learning algorithms with the new data available on the system and get predictions for ongoing courses. ''Predictions are not limited to ongoing courses, this depends on the model.''<br />
* '''Evaluate:''' Evaluate the prediction model by getting all the training data available on the site, calculating all the indicators and the target and passing the resulting dataset to machine learning backends, they will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process use all information available on the site, even if it is very old, the accuracy returned by the evaluation process will generally be lower than the real model accuracy as indicators are more reliably calculated straight after training data is available because the site state changes along time. The metric used as accuracy is the ''Matthew’s correlation coefficient'' (good metric for binary classifications)<br />
* '''Log:''' View previous evaluations log, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curves graphs, the tensorboard log dir or the model's Matthew’s correlation coefficient. The information available will depend on the machine learning backend in use.<br />
* '''Edit:''' You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.<br />
* '''Enable / Disable:''' The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.<br />
* '''Export''': Export your site training data to share it with your partner institutions.<br />
* '''Invalid site elements''': Reports on what elements in your site can not be analysed by this model<br />
* '''Clear predictions''': Clears all the model predictions and training data<br />
<br />
[[File:model-evaluation.jpeg]]<br />
<br />
=== Core models ===<br />
<br />
==== Students at risk of dropping out ====<br />
<br />
This model predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. The prediction model uses the Community of Inquiry model of student engagement, consisting of three parts:<br />
<br />
* Cognitive presence<br />
* Social presence<br />
* Teacher presence<br />
<br />
This prediction model is able to analyze and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success only in perfect copies of courses offered in the past. However, there are some limitations:<br />
<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Inspire. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script (https://github.com/moodlehq/moodle-tool_inspire/blob/master/cli/guess_course_start_and_end.php) that attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the indicator set (see below). Courses which do not include several core Moodle activities per “time slice” will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
<br />
==== No teaching ====<br />
<br />
This model insights will inform site managers of which courses with a close start date will not have teaching activity. This is a simple model and it does not use machine learning backend to return predictions. It bases the predictions on assumptions like there is no teaching if there are no students.<br />
<br />
== Predictions and Insights ==<br />
<br />
Models will start generating predictions at different point in time, depending on the site prediction models and the site courses start and end dates. <br />
<br />
Each model defines which predictions will generate insights and which predictions will be ignored. This is an example of ''Students at risk of dropping out'' prediction model; if a student is predicted as not at risk no insight is generated as what is interesting is to know which students are at risk of dropping out of courses, not which students are not at risk.<br />
<br />
[[File:prediction-model-insights.jpeg]]<br />
<br />
[[Category:Analytics]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Inspire&diff=129233Inspire2017-11-09T14:38:46Z<p>Dmonllao: </p>
<hr />
<div>{{Note|Note that this page refers to https://moodle.org/plugins/tool_inspire plugin which has been integrated in core as [Analytics]; check that page for Moodle 3.4 information}}<br />
<br />
{{Infobox plugin<br />
|type = Admin tool<br />
|entry = https://moodle.org/plugins/tool_inspire<br />
|tracker = <br />
|discussion = https://moodle.org/course/view.php?id=17233<br />
|maintainer = [[User:David Monllaó|David Monllaó]], [[Emdalton1|Elizabeth Dalton]]<br />
|float = right<br />
}}<br />
<br />
==Overview==<br />
The Inspire plugin implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.<br />
<br />
===Features===<br />
* A built-in prediction model: Students at risk of dropping out of courses.<br />
* A set of student engagement indicators based on the Community of Inquiry.<br />
* Built-in tools to evaluate models against your site's data<br />
* Proactive notifications for instructors using Events<br />
* Instructors can easily send messages to students identified by the model, or jump to the Outline report for that student for more detail about student activity<br />
* An API to build indicators and prediction models for third-party Moodle plugins<br />
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends<br />
<br />
'''Note: PHP 7.x is required.'''<br />
<br />
===Limitations===<br />
The initial 3.3-r1 version of the Inspire plugin has some limitations:<br />
<br />
* This version is installed as a third-party plugin. Future versions will become part of Moodle Core.<br />
* This version only reads activity logs from the standard log store. A log store selector will be added in future versions of the plugin.<br />
* This version requires a site with previous completed course, ideally using the Moodle course completion feature. This data is used to train the prediction engine.<br />
* The prediction model included with this version requires that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Inspire.<br />
* Models and predictions are only visible to teachers and administrators at present.<br />
<br />
We are continuing to enhance Inspire, and expanded capabilities will be released going forward. To help contribute to our progress, please join the conversation at the Project Inspire Community. In particular, we need data sets from a wide variety of Moodle-using institutions in order to be able to ship a working prediction model that does not depend on local site data before it can be used.<br />
<br />
==Installation==<br />
See [[Installing plugins#installing a plugin]]. <br />
<br />
If you are manually installing, this plugin should be unpacked to admin/tool/inspire.<br />
<br />
==Plugin configuration==<br />
<br />
After installation completes, you will be prompted to configure the settings for the Inspire plugins. You can also access these options at any time within the Site Administration menu:<br />
<br />
:Dashboard / Site administration / Plugins / Admin tools / Inspire<br />
<br />
Or at:<br />
<br />
http://your.moodle.site/moodle/admin/settings.php?section=inspiremanagement<br />
<br />
<br />
===Predictions processor===<br />
<br />
[[Image:inspire01_predictions_processor.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. This version of Inspire is shipped with 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python processor is more powerful and it generates graphs with the model performance but it requires setting up extra tools like Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodleinspire python package: <br />
$ pip install moodleinspire<br />
<br />
===Time splitting methods===<br />
<br />
[[Image:inspire15_time_splitting.png|frame|center|Time splitting methods]]<br />
<br />
The time splitting method divides each course into segments, based on the “Course start date” and “Course end date” settings per course. The predictions engine will run at the end of each segment. It is recommended that you only enable the time splitting methods you plan to use; the site contents analyser will calculate all indicators using each of the enabled time splitting methods. The more enabled time splitting methods, the slower the evaluation process will be.<br />
<br />
The use of these time splitting methods allows insights generated from one course to be used on another course, even if the two are not exactly the same length.<br />
<br />
===Models output directory===<br />
<br />
[[Image:inspire04_models_output.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where evaluation data is stored. Be sure this directory exists and is writable by the web server.<br />
===Teacher roles===<br />
<br />
[[Image:inspire05_teacher_roles.png|frame|center|Teacher roles]]<br />
<br />
Inspire needs to understand which users should be treated as “teachers” within a course or other context. The actions of teachers are handled differently from the actions of students in the prediction models.<br />
<br />
===Student roles===<br />
<br />
[[Image:inspire06_student_roles.png|frame|center|Student roles]]<br />
<br />
Inspire needs to understand which users should be treated as “students” within a course or other context. These roles will be recognized by prediction models. For example, the built-in model, “Identify students at risk,” generates predictions only for users identified as students within courses.<br />
<br />
==Model configuration==<br />
Inspire can support multiple prediction models at once. Each model requires a certain amount of configuration before use. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
Examples of prediction targets:<br />
* Student will not complete a course using defined course completion criteria<br />
* Student will not achieve a passing grade as defined in the course gradebook <br />
* Student will not attain the competencies attached to the course before the course end date<br />
* An unacceptable proportion of students in a given course will not achieve a passing grade, based on the course design<br />
* An unacceptable proportion of students in a given course will not achieve a passing grade, based on teacher actions in the course<br />
<br />
Models apply to specific contexts. For example, a model predicting student success in a course is operating in the context of a course. A model could be defined to apply to a cohort of students receiving a specific intervention, or to categories of courses (e.g. to compare courses in different academic subjects or using different teaching methods).<br />
<br />
===Models===<br />
This version of Inspire (3.3-r1) contains one model:<br />
<br />
* Students at risk of dropping out<br />
<br />
Additional models will be implemented in the future. An API is also provided for community model development.<br />
<br />
====Students at risk of dropping out====<br />
<br />
This model predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. The prediction model uses the Community of Inquiry model of student engagement, consisting of three parts:<br />
<br />
* Cognitive presence<br />
* Social presence<br />
* Teacher presence<br />
<br />
This prediction model is able to analyze and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success only in perfect copies of courses offered in the past. However, there are some limitations:<br />
<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Inspire. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script (https://github.com/moodlehq/moodle-tool_inspire/blob/master/cli/guess_course_start_and_end.php) that attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the indicator set (see below). Courses which do not include several core Moodle activities per “time slice” will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
<br />
===Indicators===<br />
In this model, indicators are defined for the cognitive presence and social presence of each of the Core activity plugins, along with a few other indicators.<br />
<br />
[[Image:inspire07_indicators.png|frame|center|Predictive indicators]]<br />
<br />
Additional indicators will be incorporated in future versions of Inspire. The API also supports community development of indicators.<br />
<br />
===The Web and CLI interfaces===<br />
Large production sites require intense, resource-heavy processing when a model is first enabled and evaluated. Inspire allows operations to be executed via a web-based interface or from the command line.<br />
<br />
* The web-based interface is appropriate for smaller sites, e.g. test or prototype sites.<br />
:Dashboard / Site administration / Reports / Inspire models<br />
* The command-line interface is appropriate for larger, production sites or copies of production sites.<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
<br />
For performance reasons, we do not recommend attempting to use the web-based interface for model configuration and evaluation on production sites.<br />
<br />
===Enable the model===<br />
Before you can evaluate a model, you must select the time splitting method and enable it.<br />
<br />
====Enable the model using the CLI====<br />
<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
$ php enable_model.php --modelid=1 --timesplitting=\"\\tool_inspire\\local\\time_splitting\\quarters\"<br />
<br />
<br />
====Enable the model using the web interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Edit menu item:<br />
<br />
[[Image:inspire14_edit_menu.png|frame|center|Edit model to enable]]<br />
<br />
Use the “Edit” Action menu to view and configure the settings for this model, and to enable it.<br />
<br />
In this screen you can choose from among the time splitting methods enabled in the plugin settings, and you can also disable (or restore) some of the default indicators.<br />
<br />
[[Image:inspire12_edit_model.png|frame|center|Enable model]]<br />
<br />
===Evaluate the model===<br />
After enabling a model with a specific time-splitting method and list of indicators, evaluate the model against your existing site data (courses whose end date is in the past). This will allow you to determine which time splitting method works best with your courses. You may wish to return to the “enabling” step to alter the timesplitting method, then re-evaluate.<br />
<br />
====Evaluate the model using the CLI====<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
$ php evaluate_model.php --modelid=1 --non-interactive<br />
<br />
Analysing the site<br />
The site contents are calculated and stored in batches, during evaluation you can stop the process at any moment, the next time you run it it will continue from the point you stopped it.<br />
!! There is no data to evaluate the model !!<br />
<br />
====Evaluate the model using the Web-based interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Evaluate menu item:<br />
<br />
[[Image:inspire02_models_list.png|frame|center|Select model to evaluate]]<br />
<br />
[[Image:inspire03_evaluate_models.png|frame|center|View model evaluations]]<br />
<br />
<br />
===Execute the model===<br />
Once the model has been evaluated against past course data, the model can be executed to begin to make predictions about current and future courses (courses with start dates in the past or future, and end dates in the future).<br />
<br />
====Execute the model using the CLI====<br />
These are the 2 tasks that will run through cron regularly. You can force their execution.<br />
<br />
php admin/tool/task/cli/schedule_task.php --execute=\\tool_inspire\\task\\train_models<br />
php admin/tool/task/cli/schedule_task.php --execute=\\tool_inspire\\task\\predict_models<br />
<br />
<br />
The “train” process reviews data from previously completed courses. The “predict” process generates predictions using current courses (or future courses that have not yet started, if the model is designed for that purpose).<br />
<br />
====Execute the model using the web interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Execute menu item:<br />
<br />
[[Image:inspire17_execute_model.png|frame|center|Select model to begin execution]]<br />
<br />
<br />
==View predictions==<br />
Predictions are viewed within the web-based interface and are automatically sent to the teacher of a course using notifications. Note that you will first need to enable and execute the model.<br />
<br />
<br />
===Access predictions as a manager/admin===<br />
Go to Site administration > Reports > Inspire models<br />
Select a context from the predictions list menu, e.g. a specific course. Available contexts depend on the model definition.<br />
<br />
[[Image:inspire09_view_predictions.png|frame|center|View predictions as manager]]<br />
<br />
===Access predictions as a teacher===<br />
Go to an ongoing course and Course administration > Reports > Insights<br />
A user with tool/inspire:listinsights capability will also automatically receive notifications when new predictions are available for them. e.g. Course teachers will receive a notification about their students at risk of dropping out.<br />
<br />
[[Image:inspire13_view_notifications.png|frame|center|View Inspire notifications]]<br />
<br />
From this screen, the teacher can view more details or send a message directly to the student:<br />
<br />
[[Image:inspire10_notifications_list.png|frame|center|View notification details]]<br />
<br />
===View prediction details===<br />
<br />
[[Image:inspire11_view_prediction_details.png|frame|center|View prediction details]]<br />
<br />
The prediction details show which of the indicator values were used in the prediction, and what the student’s values for those indicators are.<br />
<br />
Again, the teacher can send a message to the student at this point, or view the Outline report.<br />
<br />
==Developer’s documentation==<br />
Architecture overview (mostly for developers / systems administrators / researchers): https://docs.moodle.org/dev/Project_Inspire_API<br />
<br />
==How to request a new feature==<br />
* Check in the open issues if your feature have been already requested https://github.com/moodlehq/moodle-tool_inspire/issues<br />
* https://tracker.moodle.org/issues/?jql=project%20%3D%20MDL%20AND%20component%20%3D%20Analytics <br />
* If not, create a New issue Project: Moodle (MDL)<br />
* Select the component Analytics and your Moodle version<br />
* Explain your needs in the Description field <br />
* Vote and Watch the issue you have created<br />
<br />
==Credits==<br />
* David Monllaó - Lead Maintainer<br />
* Elizabeth Dalton - Research Analyst<br />
<br />
==See Also==<br />
* [https://moodle.org/plugins/tool_inspire Inspire tool in Plugins database]<br />
* [https://moodle.org/course/view.php?id=17233 Project Inspire Community]<br />
* [[Learning analytics]] documentation<br />
* [https://moodle.org/mod/forum/view.php?id=8044 Analytics and reporting forum]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Inspire&diff=129232Inspire2017-11-09T14:35:57Z<p>Dmonllao: </p>
<hr />
<div>{{Note that this page refers to https://moodle.org/plugins/tool_inspire plugin which has been integrated in core as [Analytics]; check that page for Moodle 3.4 information}}<br />
<br />
{{Infobox plugin<br />
|type = Admin tool<br />
|entry = https://moodle.org/plugins/tool_inspire<br />
|tracker = <br />
|discussion = https://moodle.org/course/view.php?id=17233<br />
|maintainer = [[User:David Monllaó|David Monllaó]], [[Emdalton1|Elizabeth Dalton]]<br />
|float = right<br />
}}<br />
<br />
==Overview==<br />
The Inspire plugin implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.<br />
<br />
===Features===<br />
* A built-in prediction model: Students at risk of dropping out of courses.<br />
* A set of student engagement indicators based on the Community of Inquiry.<br />
* Built-in tools to evaluate models against your site's data<br />
* Proactive notifications for instructors using Events<br />
* Instructors can easily send messages to students identified by the model, or jump to the Outline report for that student for more detail about student activity<br />
* An API to build indicators and prediction models for third-party Moodle plugins<br />
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends<br />
<br />
'''Note: PHP 7.x is required.'''<br />
<br />
===Limitations===<br />
The initial 3.3-r1 version of the Inspire plugin has some limitations:<br />
<br />
* This version is installed as a third-party plugin. Future versions will become part of Moodle Core.<br />
* This version only reads activity logs from the standard log store. A log store selector will be added in future versions of the plugin.<br />
* This version requires a site with previous completed course, ideally using the Moodle course completion feature. This data is used to train the prediction engine.<br />
* The prediction model included with this version requires that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Inspire.<br />
* Models and predictions are only visible to teachers and administrators at present.<br />
<br />
We are continuing to enhance Inspire, and expanded capabilities will be released going forward. To help contribute to our progress, please join the conversation at the Project Inspire Community. In particular, we need data sets from a wide variety of Moodle-using institutions in order to be able to ship a working prediction model that does not depend on local site data before it can be used.<br />
<br />
==Installation==<br />
See [[Installing plugins#installing a plugin]]. <br />
<br />
If you are manually installing, this plugin should be unpacked to admin/tool/inspire.<br />
<br />
==Plugin configuration==<br />
<br />
After installation completes, you will be prompted to configure the settings for the Inspire plugins. You can also access these options at any time within the Site Administration menu:<br />
<br />
:Dashboard / Site administration / Plugins / Admin tools / Inspire<br />
<br />
Or at:<br />
<br />
http://your.moodle.site/moodle/admin/settings.php?section=inspiremanagement<br />
<br />
<br />
===Predictions processor===<br />
<br />
[[Image:inspire01_predictions_processor.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. This version of Inspire is shipped with 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python processor is more powerful and it generates graphs with the model performance but it requires setting up extra tools like Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodleinspire python package: <br />
$ pip install moodleinspire<br />
<br />
===Time splitting methods===<br />
<br />
[[Image:inspire15_time_splitting.png|frame|center|Time splitting methods]]<br />
<br />
The time splitting method divides each course into segments, based on the “Course start date” and “Course end date” settings per course. The predictions engine will run at the end of each segment. It is recommended that you only enable the time splitting methods you plan to use; the site contents analyser will calculate all indicators using each of the enabled time splitting methods. The more enabled time splitting methods, the slower the evaluation process will be.<br />
<br />
The use of these time splitting methods allows insights generated from one course to be used on another course, even if the two are not exactly the same length.<br />
<br />
===Models output directory===<br />
<br />
[[Image:inspire04_models_output.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where evaluation data is stored. Be sure this directory exists and is writable by the web server.<br />
===Teacher roles===<br />
<br />
[[Image:inspire05_teacher_roles.png|frame|center|Teacher roles]]<br />
<br />
Inspire needs to understand which users should be treated as “teachers” within a course or other context. The actions of teachers are handled differently from the actions of students in the prediction models.<br />
<br />
===Student roles===<br />
<br />
[[Image:inspire06_student_roles.png|frame|center|Student roles]]<br />
<br />
Inspire needs to understand which users should be treated as “students” within a course or other context. These roles will be recognized by prediction models. For example, the built-in model, “Identify students at risk,” generates predictions only for users identified as students within courses.<br />
<br />
==Model configuration==<br />
Inspire can support multiple prediction models at once. Each model requires a certain amount of configuration before use. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
Examples of prediction targets:<br />
* Student will not complete a course using defined course completion criteria<br />
* Student will not achieve a passing grade as defined in the course gradebook <br />
* Student will not attain the competencies attached to the course before the course end date<br />
* An unacceptable proportion of students in a given course will not achieve a passing grade, based on the course design<br />
* An unacceptable proportion of students in a given course will not achieve a passing grade, based on teacher actions in the course<br />
<br />
Models apply to specific contexts. For example, a model predicting student success in a course is operating in the context of a course. A model could be defined to apply to a cohort of students receiving a specific intervention, or to categories of courses (e.g. to compare courses in different academic subjects or using different teaching methods).<br />
<br />
===Models===<br />
This version of Inspire (3.3-r1) contains one model:<br />
<br />
* Students at risk of dropping out<br />
<br />
Additional models will be implemented in the future. An API is also provided for community model development.<br />
<br />
====Students at risk of dropping out====<br />
<br />
This model predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. The prediction model uses the Community of Inquiry model of student engagement, consisting of three parts:<br />
<br />
* Cognitive presence<br />
* Social presence<br />
* Teacher presence<br />
<br />
This prediction model is able to analyze and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success only in perfect copies of courses offered in the past. However, there are some limitations:<br />
<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Inspire. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script (https://github.com/moodlehq/moodle-tool_inspire/blob/master/cli/guess_course_start_and_end.php) that attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the indicator set (see below). Courses which do not include several core Moodle activities per “time slice” will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
<br />
===Indicators===<br />
In this model, indicators are defined for the cognitive presence and social presence of each of the Core activity plugins, along with a few other indicators.<br />
<br />
[[Image:inspire07_indicators.png|frame|center|Predictive indicators]]<br />
<br />
Additional indicators will be incorporated in future versions of Inspire. The API also supports community development of indicators.<br />
<br />
===The Web and CLI interfaces===<br />
Large production sites require intense, resource-heavy processing when a model is first enabled and evaluated. Inspire allows operations to be executed via a web-based interface or from the command line.<br />
<br />
* The web-based interface is appropriate for smaller sites, e.g. test or prototype sites.<br />
:Dashboard / Site administration / Reports / Inspire models<br />
* The command-line interface is appropriate for larger, production sites or copies of production sites.<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
<br />
For performance reasons, we do not recommend attempting to use the web-based interface for model configuration and evaluation on production sites.<br />
<br />
===Enable the model===<br />
Before you can evaluate a model, you must select the time splitting method and enable it.<br />
<br />
====Enable the model using the CLI====<br />
<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
$ php enable_model.php --modelid=1 --timesplitting=\"\\tool_inspire\\local\\time_splitting\\quarters\"<br />
<br />
<br />
====Enable the model using the web interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Edit menu item:<br />
<br />
[[Image:inspire14_edit_menu.png|frame|center|Edit model to enable]]<br />
<br />
Use the “Edit” Action menu to view and configure the settings for this model, and to enable it.<br />
<br />
In this screen you can choose from among the time splitting methods enabled in the plugin settings, and you can also disable (or restore) some of the default indicators.<br />
<br />
[[Image:inspire12_edit_model.png|frame|center|Enable model]]<br />
<br />
===Evaluate the model===<br />
After enabling a model with a specific time-splitting method and list of indicators, evaluate the model against your existing site data (courses whose end date is in the past). This will allow you to determine which time splitting method works best with your courses. You may wish to return to the “enabling” step to alter the timesplitting method, then re-evaluate.<br />
<br />
====Evaluate the model using the CLI====<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
$ php evaluate_model.php --modelid=1 --non-interactive<br />
<br />
Analysing the site<br />
The site contents are calculated and stored in batches, during evaluation you can stop the process at any moment, the next time you run it it will continue from the point you stopped it.<br />
!! There is no data to evaluate the model !!<br />
<br />
====Evaluate the model using the Web-based interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Evaluate menu item:<br />
<br />
[[Image:inspire02_models_list.png|frame|center|Select model to evaluate]]<br />
<br />
[[Image:inspire03_evaluate_models.png|frame|center|View model evaluations]]<br />
<br />
<br />
===Execute the model===<br />
Once the model has been evaluated against past course data, the model can be executed to begin to make predictions about current and future courses (courses with start dates in the past or future, and end dates in the future).<br />
<br />
====Execute the model using the CLI====<br />
These are the 2 tasks that will run through cron regularly. You can force their execution.<br />
<br />
php admin/tool/task/cli/schedule_task.php --execute=\\tool_inspire\\task\\train_models<br />
php admin/tool/task/cli/schedule_task.php --execute=\\tool_inspire\\task\\predict_models<br />
<br />
<br />
The “train” process reviews data from previously completed courses. The “predict” process generates predictions using current courses (or future courses that have not yet started, if the model is designed for that purpose).<br />
<br />
====Execute the model using the web interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Execute menu item:<br />
<br />
[[Image:inspire17_execute_model.png|frame|center|Select model to begin execution]]<br />
<br />
<br />
==View predictions==<br />
Predictions are viewed within the web-based interface and are automatically sent to the teacher of a course using notifications. Note that you will first need to enable and execute the model.<br />
<br />
<br />
===Access predictions as a manager/admin===<br />
Go to Site administration > Reports > Inspire models<br />
Select a context from the predictions list menu, e.g. a specific course. Available contexts depend on the model definition.<br />
<br />
[[Image:inspire09_view_predictions.png|frame|center|View predictions as manager]]<br />
<br />
===Access predictions as a teacher===<br />
Go to an ongoing course and Course administration > Reports > Insights<br />
A user with tool/inspire:listinsights capability will also automatically receive notifications when new predictions are available for them. e.g. Course teachers will receive a notification about their students at risk of dropping out.<br />
<br />
[[Image:inspire13_view_notifications.png|frame|center|View Inspire notifications]]<br />
<br />
From this screen, the teacher can view more details or send a message directly to the student:<br />
<br />
[[Image:inspire10_notifications_list.png|frame|center|View notification details]]<br />
<br />
===View prediction details===<br />
<br />
[[Image:inspire11_view_prediction_details.png|frame|center|View prediction details]]<br />
<br />
The prediction details show which of the indicator values were used in the prediction, and what the student’s values for those indicators are.<br />
<br />
Again, the teacher can send a message to the student at this point, or view the Outline report.<br />
<br />
==Developer’s documentation==<br />
Architecture overview (mostly for developers / systems administrators / researchers): https://docs.moodle.org/dev/Project_Inspire_API<br />
<br />
==How to request a new feature==<br />
* Check in the open issues if your feature have been already requested https://github.com/moodlehq/moodle-tool_inspire/issues<br />
* https://tracker.moodle.org/issues/?jql=project%20%3D%20MDL%20AND%20component%20%3D%20Analytics <br />
* If not, create a New issue Project: Moodle (MDL)<br />
* Select the component Analytics and your Moodle version<br />
* Explain your needs in the Description field <br />
* Vote and Watch the issue you have created<br />
<br />
==Credits==<br />
* David Monllaó - Lead Maintainer<br />
* Elizabeth Dalton - Research Analyst<br />
<br />
==See Also==<br />
* [https://moodle.org/plugins/tool_inspire Inspire tool in Plugins database]<br />
* [https://moodle.org/course/view.php?id=17233 Project Inspire Community]<br />
* [[Learning analytics]] documentation<br />
* [https://moodle.org/mod/forum/view.php?id=8044 Analytics and reporting forum]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Inspire&diff=129230Inspire2017-11-09T14:30:16Z<p>Dmonllao: </p>
<hr />
<div>{{Notice Note that}}<br />
<br />
{{Infobox plugin<br />
|type = Admin tool<br />
|entry = https://moodle.org/plugins/tool_inspire<br />
|tracker = <br />
|discussion = https://moodle.org/course/view.php?id=17233<br />
|maintainer = [[User:David Monllaó|David Monllaó]], [[Emdalton1|Elizabeth Dalton]]<br />
|float = right<br />
}}<br />
<br />
==Overview==<br />
The Inspire plugin implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.<br />
<br />
===Features===<br />
* A built-in prediction model: Students at risk of dropping out of courses.<br />
* A set of student engagement indicators based on the Community of Inquiry.<br />
* Built-in tools to evaluate models against your site's data<br />
* Proactive notifications for instructors using Events<br />
* Instructors can easily send messages to students identified by the model, or jump to the Outline report for that student for more detail about student activity<br />
* An API to build indicators and prediction models for third-party Moodle plugins<br />
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends<br />
<br />
'''Note: PHP 7.x is required.'''<br />
<br />
===Limitations===<br />
The initial 3.3-r1 version of the Inspire plugin has some limitations:<br />
<br />
* This version is installed as a third-party plugin. Future versions will become part of Moodle Core.<br />
* This version only reads activity logs from the standard log store. A log store selector will be added in future versions of the plugin.<br />
* This version requires a site with previous completed course, ideally using the Moodle course completion feature. This data is used to train the prediction engine.<br />
* The prediction model included with this version requires that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Inspire.<br />
* Models and predictions are only visible to teachers and administrators at present.<br />
<br />
We are continuing to enhance Inspire, and expanded capabilities will be released going forward. To help contribute to our progress, please join the conversation at the Project Inspire Community. In particular, we need data sets from a wide variety of Moodle-using institutions in order to be able to ship a working prediction model that does not depend on local site data before it can be used.<br />
<br />
==Installation==<br />
See [[Installing plugins#installing a plugin]]. <br />
<br />
If you are manually installing, this plugin should be unpacked to admin/tool/inspire.<br />
<br />
==Plugin configuration==<br />
<br />
After installation completes, you will be prompted to configure the settings for the Inspire plugins. You can also access these options at any time within the Site Administration menu:<br />
<br />
:Dashboard / Site administration / Plugins / Admin tools / Inspire<br />
<br />
Or at:<br />
<br />
http://your.moodle.site/moodle/admin/settings.php?section=inspiremanagement<br />
<br />
<br />
===Predictions processor===<br />
<br />
[[Image:inspire01_predictions_processor.png|frame|center|Predictions processor selection]]<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. This version of Inspire is shipped with 2 prediction processors:<br />
<br />
* The PHP processor is the default. There are no other system requirements to use this processor.<br />
* The Python processor is more powerful and it generates graphs with the model performance but it requires setting up extra tools like Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodleinspire python package: <br />
$ pip install moodleinspire<br />
<br />
===Time splitting methods===<br />
<br />
[[Image:inspire15_time_splitting.png|frame|center|Time splitting methods]]<br />
<br />
The time splitting method divides each course into segments, based on the “Course start date” and “Course end date” settings per course. The predictions engine will run at the end of each segment. It is recommended that you only enable the time splitting methods you plan to use; the site contents analyser will calculate all indicators using each of the enabled time splitting methods. The more enabled time splitting methods, the slower the evaluation process will be.<br />
<br />
The use of these time splitting methods allows insights generated from one course to be used on another course, even if the two are not exactly the same length.<br />
<br />
===Models output directory===<br />
<br />
[[Image:inspire04_models_output.png|frame|center|Models output directory]]<br />
<br />
This setting allows you to define a directory where evaluation data is stored. Be sure this directory exists and is writable by the web server.<br />
===Teacher roles===<br />
<br />
[[Image:inspire05_teacher_roles.png|frame|center|Teacher roles]]<br />
<br />
Inspire needs to understand which users should be treated as “teachers” within a course or other context. The actions of teachers are handled differently from the actions of students in the prediction models.<br />
<br />
===Student roles===<br />
<br />
[[Image:inspire06_student_roles.png|frame|center|Student roles]]<br />
<br />
Inspire needs to understand which users should be treated as “students” within a course or other context. These roles will be recognized by prediction models. For example, the built-in model, “Identify students at risk,” generates predictions only for users identified as students within courses.<br />
<br />
==Model configuration==<br />
Inspire can support multiple prediction models at once. Each model requires a certain amount of configuration before use. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
Examples of prediction targets:<br />
* Student will not complete a course using defined course completion criteria<br />
* Student will not achieve a passing grade as defined in the course gradebook <br />
* Student will not attain the competencies attached to the course before the course end date<br />
* An unacceptable proportion of students in a given course will not achieve a passing grade, based on the course design<br />
* An unacceptable proportion of students in a given course will not achieve a passing grade, based on teacher actions in the course<br />
<br />
Models apply to specific contexts. For example, a model predicting student success in a course is operating in the context of a course. A model could be defined to apply to a cohort of students receiving a specific intervention, or to categories of courses (e.g. to compare courses in different academic subjects or using different teaching methods).<br />
<br />
===Models===<br />
This version of Inspire (3.3-r1) contains one model:<br />
<br />
* Students at risk of dropping out<br />
<br />
Additional models will be implemented in the future. An API is also provided for community model development.<br />
<br />
====Students at risk of dropping out====<br />
<br />
This model predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. The prediction model uses the Community of Inquiry model of student engagement, consisting of three parts:<br />
<br />
* Cognitive presence<br />
* Social presence<br />
* Teacher presence<br />
<br />
This prediction model is able to analyze and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success only in perfect copies of courses offered in the past. However, there are some limitations:<br />
<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Inspire. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script (https://github.com/moodlehq/moodle-tool_inspire/blob/master/cli/guess_course_start_and_end.php) that attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the indicator set (see below). Courses which do not include several core Moodle activities per “time slice” will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
<br />
===Indicators===<br />
In this model, indicators are defined for the cognitive presence and social presence of each of the Core activity plugins, along with a few other indicators.<br />
<br />
[[Image:inspire07_indicators.png|frame|center|Predictive indicators]]<br />
<br />
Additional indicators will be incorporated in future versions of Inspire. The API also supports community development of indicators.<br />
<br />
===The Web and CLI interfaces===<br />
Large production sites require intense, resource-heavy processing when a model is first enabled and evaluated. Inspire allows operations to be executed via a web-based interface or from the command line.<br />
<br />
* The web-based interface is appropriate for smaller sites, e.g. test or prototype sites.<br />
:Dashboard / Site administration / Reports / Inspire models<br />
* The command-line interface is appropriate for larger, production sites or copies of production sites.<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
<br />
For performance reasons, we do not recommend attempting to use the web-based interface for model configuration and evaluation on production sites.<br />
<br />
===Enable the model===<br />
Before you can evaluate a model, you must select the time splitting method and enable it.<br />
<br />
====Enable the model using the CLI====<br />
<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
$ php enable_model.php --modelid=1 --timesplitting=\"\\tool_inspire\\local\\time_splitting\\quarters\"<br />
<br />
<br />
====Enable the model using the web interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Edit menu item:<br />
<br />
[[Image:inspire14_edit_menu.png|frame|center|Edit model to enable]]<br />
<br />
Use the “Edit” Action menu to view and configure the settings for this model, and to enable it.<br />
<br />
In this screen you can choose from among the time splitting methods enabled in the plugin settings, and you can also disable (or restore) some of the default indicators.<br />
<br />
[[Image:inspire12_edit_model.png|frame|center|Enable model]]<br />
<br />
===Evaluate the model===<br />
After enabling a model with a specific time-splitting method and list of indicators, evaluate the model against your existing site data (courses whose end date is in the past). This will allow you to determine which time splitting method works best with your courses. You may wish to return to the “enabling” step to alter the timesplitting method, then re-evaluate.<br />
<br />
====Evaluate the model using the CLI====<br />
$ cd <moodle install directory>/admin/tool/inspire/cli<br />
$ php evaluate_model.php --modelid=1 --non-interactive<br />
<br />
Analysing the site<br />
The site contents are calculated and stored in batches, during evaluation you can stop the process at any moment, the next time you run it it will continue from the point you stopped it.<br />
!! There is no data to evaluate the model !!<br />
<br />
====Evaluate the model using the Web-based interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Evaluate menu item:<br />
<br />
[[Image:inspire02_models_list.png|frame|center|Select model to evaluate]]<br />
<br />
[[Image:inspire03_evaluate_models.png|frame|center|View model evaluations]]<br />
<br />
<br />
===Execute the model===<br />
Once the model has been evaluated against past course data, the model can be executed to begin to make predictions about current and future courses (courses with start dates in the past or future, and end dates in the future).<br />
<br />
====Execute the model using the CLI====<br />
These are the 2 tasks that will run through cron regularly. You can force their execution.<br />
<br />
php admin/tool/task/cli/schedule_task.php --execute=\\tool_inspire\\task\\train_models<br />
php admin/tool/task/cli/schedule_task.php --execute=\\tool_inspire\\task\\predict_models<br />
<br />
<br />
The “train” process reviews data from previously completed courses. The “predict” process generates predictions using current courses (or future courses that have not yet started, if the model is designed for that purpose).<br />
<br />
====Execute the model using the web interface====<br />
Navigate to the Inspire Models administration page:<br />
Dashboard / Site administration / Reports / Inspire models<br />
<br />
Select a model from the list of available models, using the Execute menu item:<br />
<br />
[[Image:inspire17_execute_model.png|frame|center|Select model to begin execution]]<br />
<br />
<br />
==View predictions==<br />
Predictions are viewed within the web-based interface and are automatically sent to the teacher of a course using notifications. Note that you will first need to enable and execute the model.<br />
<br />
<br />
===Access predictions as a manager/admin===<br />
Go to Site administration > Reports > Inspire models<br />
Select a context from the predictions list menu, e.g. a specific course. Available contexts depend on the model definition.<br />
<br />
[[Image:inspire09_view_predictions.png|frame|center|View predictions as manager]]<br />
<br />
===Access predictions as a teacher===<br />
Go to an ongoing course and Course administration > Reports > Insights<br />
A user with tool/inspire:listinsights capability will also automatically receive notifications when new predictions are available for them. e.g. Course teachers will receive a notification about their students at risk of dropping out.<br />
<br />
[[Image:inspire13_view_notifications.png|frame|center|View Inspire notifications]]<br />
<br />
From this screen, the teacher can view more details or send a message directly to the student:<br />
<br />
[[Image:inspire10_notifications_list.png|frame|center|View notification details]]<br />
<br />
===View prediction details===<br />
<br />
[[Image:inspire11_view_prediction_details.png|frame|center|View prediction details]]<br />
<br />
The prediction details show which of the indicator values were used in the prediction, and what the student’s values for those indicators are.<br />
<br />
Again, the teacher can send a message to the student at this point, or view the Outline report.<br />
<br />
==Developer’s documentation==<br />
Architecture overview (mostly for developers / systems administrators / researchers): https://docs.moodle.org/dev/Project_Inspire_API<br />
<br />
==How to request a new feature==<br />
* Check in the open issues if your feature have been already requested https://github.com/moodlehq/moodle-tool_inspire/issues<br />
* https://tracker.moodle.org/issues/?jql=project%20%3D%20MDL%20AND%20component%20%3D%20Analytics <br />
* If not, create a New issue Project: Moodle (MDL)<br />
* Select the component Analytics and your Moodle version<br />
* Explain your needs in the Description field <br />
* Vote and Watch the issue you have created<br />
<br />
==Credits==<br />
* David Monllaó - Lead Maintainer<br />
* Elizabeth Dalton - Research Analyst<br />
<br />
==See Also==<br />
* [https://moodle.org/plugins/tool_inspire Inspire tool in Plugins database]<br />
* [https://moodle.org/course/view.php?id=17233 Project Inspire Community]<br />
* [[Learning analytics]] documentation<br />
* [https://moodle.org/mod/forum/view.php?id=8044 Analytics and reporting forum]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics&diff=129229Analytics2017-11-09T14:26:01Z<p>Dmonllao: </p>
<hr />
<div>{{Tracking progress}}{{New features}}<br />
<br />
==Overview==<br />
This page describes analytics features in Moodle core. Open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success to learners and teachers.<br />
<br />
== Settings ==<br />
<br />
You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
=== Predictions processor ===<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:<br />
<br />
* The PHP one is the default. There are no other system requirements to use this processor.<br />
* The Python one is more powerful and it generates graphs with the model performance but it requires setting up extra stuff like Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend package.<br />
<br />
pip install moodlemlbackend<br />
<br />
=== Time splitting methods ===<br />
<br />
The time splitting method divides the course duration in parts, the predictions engine will run at the end of these parts. It is recommended that you only enable the time splitting methods you could be interested on using; the site contents analyser will calculate all indicators using each of the enabled time splitting methods. The more enabled time splitting methods the slower the evaluation process will be.<br />
<br />
===Models output directory===<br />
<br />
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server.<br />
<br />
== Models management ==<br />
<br />
Moodle can support multiple prediction models at once. Moodle core comes with two prediction models ''Students at risk of dropping out'' and ''No teaching'', you can add your own prediction models using [https://docs.moodle.org/dev/Analytics_API Moodle analytics API]. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.<br />
<br />
You can manage your system models from ''Site Administration > Analytics > Analytics models''.<br />
<br />
[[File:prediction-models-list.jpeg]]<br />
<br />
These are some of the actions you can perform on a model:<br />
<br />
* '''Get predictions:''' Train machine learning algorithms with the new data available on the system and get predictions for ongoing courses. ''Predictions are not limited to ongoing courses, this depends on the model.''<br />
* '''Evaluate:''' Evaluate the prediction model by getting all the training data available on the site, calculating all the indicators and the target and passing the resulting dataset to machine learning backends, they will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process use all information available on the site, even if it is very old, the accuracy returned by the evaluation process will generally be lower than the real model accuracy as indicators are more reliably calculated straight after training data is available because the site state changes along time. The metric used as accuracy is the ''Matthew’s correlation coefficient'' (good metric for binary classifications)<br />
* '''Log:''' View previous evaluations log, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curves graphs, the tensorboard log dir or the model's Matthew’s correlation coefficient. The information available will depend on the machine learning backend in use.<br />
* '''Edit:''' You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.<br />
* '''Enable / Disable:''' The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.<br />
* '''Export''': Export your site training data to share it with your partner institutions.<br />
* '''Invalid site elements''': Reports on what elements in your site can not be analysed by this model<br />
* '''Clear predictions''': Clears all the model predictions and training data<br />
<br />
[[File:model-evaluation.jpeg]]<br />
<br />
=== Core models ===<br />
<br />
==== Students at risk of dropping out ====<br />
<br />
This model predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. The prediction model uses the Community of Inquiry model of student engagement, consisting of three parts:<br />
<br />
* Cognitive presence<br />
* Social presence<br />
* Teacher presence<br />
<br />
This prediction model is able to analyze and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success only in perfect copies of courses offered in the past. However, there are some limitations:<br />
<br />
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Inspire. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script (https://github.com/moodlehq/moodle-tool_inspire/blob/master/cli/guess_course_start_and_end.php) that attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.<br />
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the indicator set (see below). Courses which do not include several core Moodle activities per “time slice” will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.<br />
<br />
==== No teaching ====<br />
<br />
This model insights will inform site managers of which courses with a close start date will not have teaching activity. This is a simple model and it does not use machine learning backend to return predictions. It bases the predictions on assumptions like there is no teaching if there are no students.<br />
<br />
== Predictions and Insights ==<br />
<br />
Models will start generating predictions at different point in time, depending on the site prediction models and the site courses start and end dates. <br />
<br />
Each model defines which predictions will generate insights and which predictions will be ignored. This is an example of ''Students at risk of dropping out'' prediction model; if a student is predicted as not at risk no insight is generated as what is interesting is to know which students are at risk of dropping out of courses, not which students are not at risk.<br />
<br />
[[File:prediction-model-insights.jpeg]]<br />
<br />
[[Category:Analytics]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=File:prediction-model-insights.jpeg&diff=129215File:prediction-model-insights.jpeg2017-11-09T09:53:31Z<p>Dmonllao: Dmonllao uploaded a new version of File:prediction-model-insights.jpeg</p>
<hr />
<div></div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=File:model-evaluation.jpeg&diff=129214File:model-evaluation.jpeg2017-11-09T09:53:14Z<p>Dmonllao: Dmonllao uploaded a new version of File:model-evaluation.jpeg</p>
<hr />
<div></div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=File:prediction-models-list.jpeg&diff=129213File:prediction-models-list.jpeg2017-11-09T09:52:52Z<p>Dmonllao: Dmonllao uploaded a new version of File:prediction-models-list.jpeg</p>
<hr />
<div></div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=File:prediction-model-insights.jpeg&diff=129212File:prediction-model-insights.jpeg2017-11-09T09:48:08Z<p>Dmonllao: </p>
<hr />
<div></div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=File:model-evaluation.jpeg&diff=129211File:model-evaluation.jpeg2017-11-09T09:47:12Z<p>Dmonllao: </p>
<hr />
<div></div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=File:prediction-models-list.jpeg&diff=129210File:prediction-models-list.jpeg2017-11-09T09:47:02Z<p>Dmonllao: </p>
<hr />
<div></div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Analytics&diff=129209Analytics2017-11-09T09:45:19Z<p>Dmonllao: </p>
<hr />
<div>''Info to be added soon...''<br />
This page describes the Analytics models tool used to visualise, manage and evaluate prediction models.<br />
<br />
== Settings ==<br />
<br />
You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.<br />
<br />
=== Predictions processor ===<br />
<br />
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. This plugin is shipped with 2 prediction processors:<br />
<br />
* The PHP one is the default<br />
* The Python one is more powerful and it generates graphs with the model performance but it requires setting up extra stuff like Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend package.<br />
<br />
pip install moodlemlbackend<br />
<br />
=== Time splitting methods ===<br />
<br />
The time splitting method divides the course duration in parts, the predictions engine will run at the end of these parts. It is recommended that you only enable the time splitting methods you could be interested on using; the site contents analyser will calculate all indicators using each of the enabled time splitting methods. The more enabled time splitting methods the slower the evaluation process will be.<br />
<br />
== Models management ==<br />
<br />
You can access the tool from ''Site Administration > Analytics > Analytics models'' to see the list of prediction models.<br />
<br />
[[File:prediction-models-list.jpeg]]<br />
<br />
These are some of the actions you can perform on a model:<br />
<br />
* ''Edit:'' You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.<br />
* ''Enable / Disable:'' The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.<br />
* ''Evaluate:'' Evaluate the prediction model by getting all the training data available on the site, calculating all the indicators and the target and passing the resulting dataset to machine learning backends, they will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process use all information available on the site, even if it is very old, the accuracy returned by the evaluation process will generally be lower than the real model accuracy as indicators are more reliably calculated straight after training data is available because the site state changes along time. The metric used as accuracy is the ''Matthew’s correlation coefficient'' (good metric for binary classifications)<br />
* ''Log:'' View previous evaluations log, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curves graphs, the tensorboard log dir or the model's Matthew’s correlation coefficient. The information available will depend on the machine learning backend in use.<br />
* ''Get predictions:'' Train machine learning algorithms with the new data available on the system and get predictions for ongoing courses. ''Predictions are not limited to ongoing courses, this depends on the model.''<br />
<br />
[[File:model-evaluation.jpeg]]<br />
<br />
== Predictions and Insights ==<br />
<br />
Models will start generating predictions at different point in time, depending on the site prediction models and the site courses start and end dates. <br />
<br />
Each model defines which predictions will generate insights and which predictions will be ignored. This is an example of ''Students at risk of dropping out'' prediction model; if a student is predicted as not at risk no insight is generated as what is interesting is to know which students are at risk of dropping out of courses, not which students are not at risk.<br />
<br />
[[File:prediction-model-insights.jpeg]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=124195Global search2016-06-23T02:01:08Z<p>Dmonllao: /* The Solr server */</p>
<hr />
<div>{{More features}}<br />
{{New features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator and a search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==What can I search for?==<br />
You can search for your courses, activities information and some activities contents like forum posts, book chapters, glossary entries or collaborative wikis pages. We will add more contents in future Moodle versions. Developers interested in adding plugin's contents to global search may be interested in looking at the [https://docs.moodle.org/dev/Search_API Search API].<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
===Solr search query features===<br />
<br />
You can improve your search using any of the following search query features:<br />
<br />
* Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title<br />
* Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords<br />
* Wildcard characters ('*' or '?' ) to represent characters in the search query<br />
* Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)<br />
* Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.<br />
<br />
Note: The above features are only available when using the Solr search engine.<br />
<br />
==How is it set up?==<br />
<br />
{{Note|Global search needs a search engine. Instructions for the installing the PHP Solr extension and a Solr server are below. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own. It's recommended you set everything up first, index the site contents ''and only then'' enable Global search.}}<br />
<br />
===Setting up Solr===<br />
<br />
====General Setup====<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# Index your site's data going to ''Site administration > Reports > Global search info'' or from the CLI running '''php admin/tool/task/cli/schedule_task.php --execute="\core\task\search_index_task"''' script, being the CLI script the recommended option for big sites.<br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
====File Indexing====<br />
<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the [https://tika.apache.org/ Tika] engine which comes bundled with Solr. To enable this feature:<br />
# In ''Site administration > Plugins > Search > Solr'' enable the checkbox '''Enable file indexing'''<br />
# Set '''Maximum file size to index (kB)''' to some value - the default is '''2097152''' (2GB)<br />
#* Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.<br />
<br />
===How to install Solr===<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr](http://pecl.php.net/package/solr) The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
==== Linux (Debian/Ubuntu) ====<br />
<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
==== OSX using macports ====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
==== OSX using homebrew ====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
==== Windows ====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
==== The Solr server ====<br />
<br />
Moodle 3.1 supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released. <br />
<br />
Note that for medium/large sites you may need to increase maxBooleanClauses setting. In [https://tracker.moodle.org/browse/MDL-54992 MDL-54992] we are working on an alternative way to query the server. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it.<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.4.1<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
=====Solr 5/6 schema setup=====<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
=====Solr 4 schema setup=====<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
=====Memory and File indexing considerations=====<br />
<br />
For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, eg aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the ''-m'' option: <br />
solr start -m 12g<br />
<br />
See the documents for your version of Solr on how to increases memory when Solr is started automatically.<br />
<br />
==See also==<br />
<br />
* [https://moodle.org/mod/forum/discuss.php?d=334938 Global search doesn't index plugins?] forum discussion<br />
<br />
[[es:Búsqueda global]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=124138Global search2016-06-20T02:38:49Z<p>Dmonllao: /* What is Global search? */</p>
<hr />
<div>{{More features}}<br />
{{New features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator and a search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==What can I search for?==<br />
You can search for your courses, activities information and some activities contents like forum posts, book chapters, glossary entries or collaborative wikis pages. We will add more contents in future Moodle versions. Developers interested in adding plugin's contents to global search may be interested in looking at the [https://docs.moodle.org/dev/Search_API Search API].<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
===Solr search query features===<br />
<br />
You can improve your search using any of the following search query features:<br />
<br />
* Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title<br />
* Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords<br />
* Wildcard characters ('*' or '?' ) to represent characters in the search query<br />
* Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)<br />
* Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.<br />
<br />
Note: The above features are only available when using the Solr search engine.<br />
<br />
==How is it set up?==<br />
<br />
{{Note|Global search needs a search engine. Instructions for the installing the PHP Solr extension and a Solr server are below. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own. It's recommended you set everything up first, index the site contents ''and only then'' enable Global search.}}<br />
<br />
===Setting up Solr===<br />
<br />
====General Setup====<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# Index your site's data going to ''Site administration > Reports > Global search info'' or from the CLI running '''php admin/tool/task/cli/schedule_task.php --execute="\core\task\search_index_task"''' script, being the CLI script the recommended option for big sites.<br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
====File Indexing====<br />
<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the [https://tika.apache.org/ Tika] engine which comes bundled with Solr. To enable this feature:<br />
# In ''Site administration > Plugins > Search > Solr'' enable the checkbox '''Enable file indexing'''<br />
# Set '''Maximum file size to index (kB)''' to some value - the default is '''2097152''' (2GB)<br />
#* Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.<br />
<br />
===How to install Solr===<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr](http://pecl.php.net/package/solr) The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
==== Linux (Debian/Ubuntu) ====<br />
<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
==== OSX using macports ====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
==== OSX using homebrew ====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
==== Windows ====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
==== The Solr server ====<br />
<br />
Moodle 3.1 supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it.<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.4.1<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
=====Solr 5/6 schema setup=====<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
=====Solr 4 schema setup=====<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
=====Memory and File indexing considerations=====<br />
<br />
For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, eg aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the ''-m'' option: <br />
solr start -m 12g<br />
<br />
See the documents for your version of Solr on how to increases memory when Solr is started automatically.<br />
<br />
==See also==<br />
<br />
* [https://moodle.org/mod/forum/discuss.php?d=334938 Global search doesn't index plugins?] forum discussion<br />
<br />
[[es:Búsqueda global]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=123700Global search2016-05-20T02:02:51Z<p>Dmonllao: /* Linux (Debian/Ubuntu) */</p>
<hr />
<div>{{More features}}<br />
{{New features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator and a search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
===Solr search query features===<br />
<br />
You can improve your search using any of the following search query features:<br />
<br />
* Specifying the field to be searched by prefixing the search query with 'title:', 'content:', 'name:', or 'intro:' e.g.'title:news' returns results with the word 'news' in the title<br />
* Boolean operators ('AND', 'OR', 'NOT') to combine or exclude keywords<br />
* Wildcard characters ('*' or '?' ) to represent characters in the search query<br />
* Proximity searches ('~') e.g. mood~2 returns "moodle" (2 letters away from "mood"), Moodle Australia~3 returns results containing "Moodle HQ in Perth, Australia" (the queried terms were within 3 words of each other)<br />
* Boosting terms ('^') to boost certain words or phrases e.g. "Perth Australia"^5 "Australia" will return results with the phrase "Perth Australia" as more relevant.<br />
<br />
Note: The above features are only available when using the Solr search engine.<br />
<br />
==How is it set up?==<br />
<br />
{{Note|Global search needs a search engine. Instructions for the installing the PHP Solr extension and a Solr server are below. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own. It's recommended you set everything up first, index the site contents ''and only then'' enable Global search.}}<br />
<br />
===Setting up Solr===<br />
<br />
====General Setup====<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# Index your site's data going to ''Site administration > Reports > Global search info'' or from the CLI running '''php admin/tool/task/cli/schedule_task.php --execute="\core\task\search_index_task"''' script, being the CLI script the recommended option for big sites.<br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
====File Indexing====<br />
<br />
Solr has the ability to index the contents of files, such as File resources and attachments to Forum posts. This uses the [https://tika.apache.org/ Tika] engine which comes bundled with Solr. To enable this feature:<br />
# In ''Site administration > Plugins > Search > Solr'' enable the checkbox '''Enable file indexing'''<br />
# Set '''Maximum file size to index (kB)''' to some value - the default is '''2097152''' (2GB)<br />
#* Files larger than this limit will not be sent for Solr for indexing, but the file name will still be indexed.<br />
<br />
===How to install Solr===<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr](http://pecl.php.net/package/solr) The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
==== Linux (Debian/Ubuntu) ====<br />
<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
sudo service apache2 restart<br />
<br />
==== OSX using macports ====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
==== OSX using homebrew ====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
==== Windows ====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
==== The Solr server ====<br />
<br />
Moodle 3.1 supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it.<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.4.1<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
=====Solr 5/6 schema setup=====<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
=====Solr 4 schema setup=====<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
=====Memory and File indexing considerations=====<br />
<br />
For large sites, and particularly if you are using the File indexing feature, it's a good idea to give Solr plenty of memory, eg aound 10-20GB. To start Solr with more than its default 512MB of RAM, use the ''-m'' option: <br />
solr start -m 12g<br />
<br />
See the documents for your version of Solr on how to increases memory when Solr is started automatically.<br />
<br />
[[es:Búsqueda global]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=123552Global search2016-05-17T01:55:28Z<p>Dmonllao: /* How is it set up? */</p>
<hr />
<div>{{New features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator and a search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch2.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2a.png|thumb|center|600px|Search results]]<br />
<br />
==How is it set up?==<br />
<br />
{{Note|Global search needs a search engine. Instructions for the installing the PHP Solr extension and a Solr server are below. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own.}}<br />
<br />
You can start by enabling Global search feature but we recommend to set everything up first, index the site contents and once the process finishes enable Global search, so users can start using the feature straight after enabling it.<br />
<br />
# Set the feature up in ''Site administration > Plugins > Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr) <br />
# If you are using Solr with SSL encryption, you will need to configure Moodle as such. This is important if solr is not installed in the web server server.<br />
## You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
## Go to ''Site administration > Plugins > Search > Solr''<br />
## Set '''Secure mode''' to Yes<br />
## '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
## '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
## '''SSL key Password''' to The password used to lock the SSL Key<br />
## '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
# Index your site's data going to ''Site administration > Reports > Global search info'' or from the CLI running '''php admin/tool/task/cli/schedule_task.php --execute="\core\task\search_index_task"''' script, being the CLI script the recommended option for big sites.<br />
# Enable Global search in ''Site administration > Advanced features''<br />
<br />
===How to install Solr===<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr](http://pecl.php.net/package/solr) The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
==== Linux (Debian/Ubuntu) ====<br />
<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo service apache2 restart<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
<br />
==== OSX using macports ====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
==== OSX using homebrew ====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
==== Windows ====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
==== The Solr server ====<br />
<br />
Moodle 3.1 supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it.<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.4.1<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
=====Solr 5/6 schema setup=====<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields. You will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
=====Solr 4 schema setup=====<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
<br />
[[es:Búsqueda global]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Global_search&diff=123414Global search2016-05-13T00:14:56Z<p>Dmonllao: /* The Solr server */</p>
<hr />
<div>{{New features}}<br />
==What is Global search?==<br />
*Global search lets you search everywhere on the Moodle site that you have access to. A student can search their courses for particular lecture notes, for example, or a teacher could search for subject-related activities.<br />
*The feature needs to be enabled by the administrator and a search box is then available next to the user menu.<br />
<br />
[[File:NFGlobalsearchbox.png|thumb|center|480px|Searching the site (when global search is enabled)]]<br />
<br />
*A [[Global search block]] may also be added to course pages.<br />
<br />
==How does it work?==<br />
*Click the search icon by the user menu and type a search term into the box that appears, or type into the box in the [[Global search block]] if enabled.<br />
*On the next screen, you can simply click the search button to search everywhere, or expand the Filter to search in specific areas:<br />
<br />
[[File:NFFilteringGlobalSearch.png|thumb|center|600px|Filtering the search]]<br />
<br />
*You will then see results displayed from all areas of Moodle you have access to:<br />
<br />
[[File:31GSResults2.png|thumb|center|600px|Search results]]<br />
<br />
==How is it set up?==<br />
<br />
{{Note|Global search needs a search engine. Instructions for the installing the PHP Solr extension and a Solr server are below. See the [https://docs.moodle.org/dev/Search_engines Developer docs on Search engine plugins] if you wish to write your own.}}<br />
<br />
# Enable Global search in ''Site administration > Advanced features''<br />
# Set the feature up in ''Site administration > Plugins >Search > Manage global search'' by selecting Solr as the search engine and ticking search area checkboxes as required<br />
# In ''Site administration > Plugins > Search > Solr'', set '''Host name''' to localhost, '''Port''' to 8983 and '''Index name''' to 'moodle' (the name of the index in Solr)<br />
<br />
If you are using Solr with SSL encryption, you will need to configure Moodle as such.<br />
# You will need a separate key file and cacert file, both in pem format, located on your server Moodle, and readable by the PHP process.<br />
# Go to ''Site administration > Plugins > Search > Solr''<br />
# Set '''Secure mode''' to Yes<br />
# '''SSL certificate''' to /path/to/certs/solr-ssl.cacert.pem<br />
# '''SSL key''' to /path/to/certs/solr-ssl.key.pem<br />
# '''SSL key Password''' to The password used to lock the SSL Key<br />
# '''SSL CA certificates name''' to /path/to/certs/solr-ssl.cacert.pem<br />
<br />
<br />
===How to install Solr===<br />
<br />
You need PHP Solr extension installed. You can download the official latest versions from [http://pecl.php.net/package/solr](http://pecl.php.net/package/solr) The minimum required version is PECL Solr 2.1 for PHP 5 branch and PECL Solr 2.4 for PHP 7 branch.<br />
<br />
Basic installation steps (using apache web server):<br />
<br />
==== Linux (Debian/Ubuntu) ====<br />
<br />
sudo apt-get install libpcre3-dev libxml2-dev libcurl4-openssl-dev<br />
sudo apt-get install php5-dev<br />
sudo apt-get install php-pear<br />
sudo pecl install solr<br />
sudo service apache2 restart<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/apache2/conf.d/solr.ini"<br />
sudo sh -c "echo 'extension=solr.so' > /etc/php5/cli/conf.d/solr.ini"<br />
<br />
==== OSX using macports ====<br />
<br />
sudo port install apache-solr4<br />
sudo port install php54-solr<br />
<br />
==== OSX using homebrew ====<br />
<br />
brew install homebrew/php/php56-solr<br />
<br />
==== Windows ====<br />
<br />
Install the pecl package as usual. (This has not yet been tested.)<br />
<br />
==== The Solr server ====<br />
<br />
Moodle 3.1 supports Solr server from 4.0 onwards, although you can only use the Solr schema setup script that we provide with Moodle from Solr 5. The latest Solr 5 available version is the recommended one; the same will apply to Solr 6 once it is released. <br />
<br />
The following example snippet (feel free to copy & paste into a .sh script with execution permissions) will download Solr 5.4.1 (replace it with latest 5.x) in the current directory, start the solr server and create an index in it named '''moodle''' to add moodle data to it.<br />
<br />
#!/bin/bash<br />
set -e<br />
SOLRVERSION=5.4.1<br />
SOLRNAME=solr-$SOLRVERSION<br />
SOLRTAR=$SOLRNAME'.tgz'<br />
INDEXNAME=moodle<br />
if [ -d $SOLRNAME ]; then<br />
echo "Error: Directory $SOLRNAME already exists, remove it before starting the setup again."<br />
exit 1<br />
fi<br />
if [ ! -f $SOLRTAR ]; then<br />
wget http://www-us.apache.org/dist/lucene/solr/$SOLRVERSION/$SOLRTAR<br />
fi<br />
tar -xvzf $SOLRTAR<br />
cd $SOLRNAME<br />
bin/solr start<br />
bin/solr create -c $INDEXNAME<br />
# After setting it up and creating the index use:<br />
# - "/yourdirectory/solrdir/bin/solr start" from CLI to start the server<br />
# - "/yourdirectory/solrdir/bin/solr stop" from CLI to stop the server.<br />
<br />
=====Solr 5/6 schema setup=====<br />
<br />
Moodle will use Solr's managed schema interface to install the required fields, you will be directed on what to do from the Manage global search page.<br />
<br />
For very large or busy sites, it is recommended that you manually remove the '''_text_''' field, and associated copy directive from, the default Solr schema. This field is not used by Moodle, and will significantly slow indexing, and increase the size of the resulting Solr core.<br />
<br />
=====Solr 4 schema setup=====<br />
<br />
You cannot use the schema setup script when using a Solr 4 server. If you really want to use the Solr 4x branch, here are the field types descriptions:<br />
<br />
Extracted from search/classes/document.php<br />
<br />
{| class="nicetable"<br />
|-<br />
! Field name<br />
! Field type<br />
! Stored<br />
! Indexed<br />
! Query field<br />
|-<br />
| id<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| false<br />
| false<br />
|-<br />
| itemid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| title<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| content<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| contextid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| areaid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| type<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| courseid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| owneruserid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| modified<br />
| org.apache.solr.schema.TrieDateField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| userid<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| description1<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| description2<br />
| org.apache.solr.schema.TextField<br />
| true<br />
| true<br />
| true<br />
|-<br />
| solr_filegroupingid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileid<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontenthash<br />
| org.apache.solr.schema.StrField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_fileindexstatus<br />
| org.apache.solr.schema.TrieIntField<br />
| true<br />
| true<br />
| false<br />
|-<br />
| solr_filecontent<br />
| org.apache.solr.schema.TextField<br />
| false<br />
| true<br />
| true<br />
|}<br />
<br />
<br />
[[es:Búsqueda global]]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Using_web_services&diff=118464Using web services2015-05-07T02:36:20Z<p>Dmonllao: </p>
<hr />
<div>{{Web services}}<br />
This document explains how an administrator can set up a web service for users to access a service. Each user will have a specific and unique security key (also known as a "token") to access the service.<br />
<br />
== Enabling web services==<br />
[[Image:Enable_web_service.jpg|thumb|Enabling web services]]<br />
# Access ''Administration > Site administration > Advanced features''<br />
# Check 'Enable web services' then click 'Save Changes'<br />
<br />
Note: For security reasons, web services should only be enabled if you intend to make use of it.<br />
<br />
== Enabling protocols ==<br />
[[Image:Enable_protocol.jpg|thumb|left|Enabling protocols]]<br />
Usually external applications that users wish to use dictate which protocols should be enabled. <br />
<br />
# Access ''Administration > Site administration > Plugins > Web services > Manage protocols''<br />
# Enable the protocols (SOAP, REST, XMLRPC, AMF, ...) as required<br />
<br />
== Enabling web service function documentation ==<br />
[[Image:Security keys and documentation.jpg|thumb|Security keys page with documentation link]]<br />
Enabling web service function documentation (also on the Manage protocols page) results in user-specific web service documentation being available for each user on their [[Security keys]] page. This option is mainly useful to web service client developers. If nobody is creating a web service client, there is no need to enable this feature.<br />
<br />
== Creating a service ==<br />
If none of the pre-build web services match your needs, you can create a custom service i.e. select which of the standard web service functions are available via that service.<br />
<br />
You can enable only the specific functions that you need to expose, so not compromising on security.<br />
[[Image:Create a service.jpg|thumb|Creating a service|left]]<br />
# Access ''Administration > Site administration > Plugins > Web services > External services''<br />
# Click Add new custom service<br />
#* 'Authorised users only' - If enabled, you will need to select the authorised users manually. Otherwise all users with appropriate permissions are allowed<br />
#* 'Required capability' - If enabled, any user accessing the web service will be checked against this selected capability. (This is just an additional optional security layer.)<br />
# Enter a name and check Enabled<br />
# Click the button 'Add service'<br />
<br />
==Adding functions to the service==<br />
[[Image:Select a web service function.jpg|thumb|Adding functions to the service]]Your service is currently empty and doesn't do anything. Web service functions need to be added. Your choice will be dictated by what you allow the external application to do. For this example, select 'Create group'.<br />
<br />
# Click 'Add functions' link<br />
# Select 'create group' function and click the 'Add functions' button<br />
<br />
''Note that deprecated functions can not be added to services although the ones that are already part of a service can remain there until they are removed from Moodle codebase.''<br />
<br />
You should be back to the service functions list. 'Required capabilities' are indicated for each function. Users need the required capabilities to run a function. The function descriptions in the API Documentation can also give you more information about the required capabilities (''Administration > Site administration > Plugins > Web services > API Documentation'').<br />
<br />
==Enabling capabilities==<br />
<br />
The final step is to grant appropriate permissions. The following capabilities should be allowed:<br />
<br />
* [[Capabilities/moodle/webservice:createtoken|moodle/webservice:createtoken]] - for allowing users to generate a security key<br />
* [[Capabilities/webservice/rest:use | webservice/rest:use]], [[Capabilities/webservice/soap:use | webservice/soap:use]], [[Capabilities/webservice/xmlrpc:use | webservice/xmlrpc:use]], [[Capabilities/webservice/amf:use | webservice/amf:use]] which match the enabled protocols.<br />
* The service ''Required capability'' if set (''Administration > Site administration > Plugins > Web services > Manage services >'''Edit''' link'').<br />
* The required capabilities for the web service functions. These required capabilities are listed when you add a function to the service. For more information about roles and capabilities, read the [[Manage roles]] documentation.<br />
<br />
Once done, the web service should be set up. Users should be able to [[Security keys|obtain a personal security key]].<br />
<br />
= Alternative settings = <br />
<br />
== Authorise only specific users==<br />
[[Image:Authorised users link.jpg|thumb]] [[Image:Authorised user selection page.jpg|thumb]]<br />
# ''Administration > Site Administration > Plugins > Web services > External Services''<br />
# Select '''Authorised users''' link (the service must have been set as '''Authorised users only''' in the '''Edit''' link)<br />
# Select some users and click '''Add'''<br />
<br />
Moodle indicates if some capabilities need to be assigned to an authorised user. Moreover if you click on the authorised user fullname, you can set up some specific options: ''IP restriction'' and ''Valid until''.<br />
<br />
== Create a token ==<br />
[[Image:Create_token.jpg|thumb]]<br />
This feature allows you to create a token for specific user. It can be useful if a user doesn't have the moodle/create:token capability. This is also the only way to create a token for an administrator. For security reason, tokens are not automatically generated in the administrator security keys page.<br />
<br />
# ''Administration > Site Administration > Plugins > Web services > Manage tokens''<br />
# Click on '''Add'''<br />
# Select the created user and service<br />
# Click on '''Saves changes'''<br />
<br />
As you created a token for this user, you do not need to assign "''moodle/webservice:createtoken''" to him/her.<br />
Finally, note that, as for authorised users, you can also set ''IP restriction'' and ''Valid until'' on a token.<br />
<br />
== See also ==<br />
<br />
*[http://www.joomdle.com/wiki/Preparing_Moodle_20#Setting_up_Moodle_Web_services Joomdle documentation about setting Moodle web services]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Using_web_services&diff=118463Using web services2015-05-07T02:34:45Z<p>Dmonllao: /* Adding functions to the service */</p>
<hr />
<div>{{Web services}}<br />
This document explains how an administrator can set up a web service for users to access a service. Each user will have a specific and unique security key (also known as a "token") to access the service.<br />
<br />
== Enabling web services==<br />
[[Image:Enable_web_service.jpg|thumb|Enabling web services]]<br />
# Access ''Administration > Site administration > Advanced features''<br />
# Check 'Enable web services' then click 'Save Changes'<br />
<br />
Note: For security reasons, web services should only be enabled if you intend to make use of it.<br />
<br />
== Enabling protocols ==<br />
[[Image:Enable_protocol.jpg|thumb|left|Enabling protocols]]<br />
Usually external applications that users wish to use dictate which protocols should be enabled. <br />
<br />
# Access ''Administration > Site administration > Plugins > Web services > Manage protocols''<br />
# Enable the protocols (SOAP, REST, XMLRPC, AMF, ...) as required<br />
<br />
== Enabling web service function documentation ==<br />
[[Image:Security keys and documentation.jpg|thumb|Security keys page with documentation link]]<br />
Enabling web service function documentation (also on the Manage protocols page) results in user-specific web service documentation being available for each user on their [[Security keys]] page. This option is mainly useful to web service client developers. If nobody is creating a web service client, there is no need to enable this feature.<br />
<br />
== Creating a service ==<br />
If none of the pre-build web services match your needs, you can create a custom service i.e. select which of the standard web service functions are available via that service.<br />
<br />
You can enable only the specific functions that you need to expose, so not compromising on security.<br />
[[Image:Create a service.jpg|thumb|Creating a service|left]]<br />
# Access ''Administration > Site administration > Plugins > Web services > External services''<br />
# Click Add new custom service<br />
#* 'Authorised users only' - If enabled, you will need to select the authorised users manually. Otherwise all users with appropriate permissions are allowed<br />
#* 'Required capability' - If enabled, any user accessing the web service will be checked against this selected capability. (This is just an additional optional security layer.)<br />
# Enter a name and check Enabled<br />
# Click the button 'Add service'<br />
<br />
==Adding functions to the service==<br />
[[Image:Select a web service function.jpg|thumb|Adding functions to the service]]Your service is currently empty and doesn't do anything. Web service functions need to be added. Your choice will be dictated by what you allow the external application to do. For this example, select 'Create group'.<br />
<br />
# Click 'Add functions' link<br />
# Select 'create group' function and click the 'Add functions' button<br />
<br />
''Note that, from Moodle 2.9, deprecated functions can not be added to services although deprecated functions that are already part of a service can remain there until they are removed from Moodle codebase.''<br />
<br />
You should be back to the service functions list. 'Required capabilities' are indicated for each function. Users need the required capabilities to run a function. The function descriptions in the API Documentation can also give you more information about the required capabilities (''Administration > Site administration > Plugins > Web services > API Documentation'').<br />
<br />
==Enabling capabilities==<br />
<br />
The final step is to grant appropriate permissions. The following capabilities should be allowed:<br />
<br />
* [[Capabilities/moodle/webservice:createtoken|moodle/webservice:createtoken]] - for allowing users to generate a security key<br />
* [[Capabilities/webservice/rest:use | webservice/rest:use]], [[Capabilities/webservice/soap:use | webservice/soap:use]], [[Capabilities/webservice/xmlrpc:use | webservice/xmlrpc:use]], [[Capabilities/webservice/amf:use | webservice/amf:use]] which match the enabled protocols.<br />
* The service ''Required capability'' if set (''Administration > Site administration > Plugins > Web services > Manage services >'''Edit''' link'').<br />
* The required capabilities for the web service functions. These required capabilities are listed when you add a function to the service. For more information about roles and capabilities, read the [[Manage roles]] documentation.<br />
<br />
Once done, the web service should be set up. Users should be able to [[Security keys|obtain a personal security key]].<br />
<br />
= Alternative settings = <br />
<br />
== Authorise only specific users==<br />
[[Image:Authorised users link.jpg|thumb]] [[Image:Authorised user selection page.jpg|thumb]]<br />
# ''Administration > Site Administration > Plugins > Web services > External Services''<br />
# Select '''Authorised users''' link (the service must have been set as '''Authorised users only''' in the '''Edit''' link)<br />
# Select some users and click '''Add'''<br />
<br />
Moodle indicates if some capabilities need to be assigned to an authorised user. Moreover if you click on the authorised user fullname, you can set up some specific options: ''IP restriction'' and ''Valid until''.<br />
<br />
== Create a token ==<br />
[[Image:Create_token.jpg|thumb]]<br />
This feature allows you to create a token for specific user. It can be useful if a user doesn't have the moodle/create:token capability. This is also the only way to create a token for an administrator. For security reason, tokens are not automatically generated in the administrator security keys page.<br />
<br />
# ''Administration > Site Administration > Plugins > Web services > Manage tokens''<br />
# Click on '''Add'''<br />
# Select the created user and service<br />
# Click on '''Saves changes'''<br />
<br />
As you created a token for this user, you do not need to assign "''moodle/webservice:createtoken''" to him/her.<br />
Finally, note that, as for authorised users, you can also set ''IP restriction'' and ''Valid until'' on a token.<br />
<br />
== See also ==<br />
<br />
*[http://www.joomdle.com/wiki/Preparing_Moodle_20#Setting_up_Moodle_Web_services Joomdle documentation about setting Moodle web services]</div>Dmonllaohttps://docs.moodle.org/310/en/index.php?title=Administration_via_command_line&diff=118462Administration via command line2015-05-07T01:44:16Z<p>Dmonllao: </p>
<hr />
<div>{{Installing Moodle}}<br />
==Running CLI scripts==<br />
If you have shell access to your web server, you may find various CLI (command line interface) scripts useful during Moodle administration. Core admin CLI tools are located in the <code>admin/cli/*</code> folder. Other plugins provide their CLI functionality via scripts in their own cli folder. For example, the enrol_db sync script is located in <code>enrol/db/cli/</code>.<br />
<br />
To avoid problems with access control, you should run them as the owner of the web server process. It is especially important for CLI installation and upgrade as they create new files in moodledata directory and the web server has to have write access to them. In Linux distributions, the user that runs the web server is usually apache or wwrun or httpd or something similar. As a root, you will probably want to execute Moodle CLI scripts like this:<br />
<br />
$ cd /path/to/your/moodle/dir<br />
$ sudo -u apache /usr/bin/php admin/cli/somescript.php --params<br />
<br />
Most of the scripts accept common --help (or -h) parameter to display the full usage information, for example:<br />
<br />
$ sudo -u apache /usr/bin/php admin/cli/install.php --help<br />
<br />
== Upgrading ==<br />
<br />
Moodle can be upgraded from the command line. As with the installation script, there is either interactive or non-interactive mode of the upgrade. The script itself does not put the site into the maintenance mode, you have to do it on your own. Also, the script does not backup any data (if you read this page, you probably have some own scripts to backup your moodledata and the database, right?)<br />
<br />
$ sudo -u apache /usr/bin/php admin/cli/upgrade.php<br />
<br />
Upgrading via command line is a very comfortable way of Moodle upgrade if you use Git checkout of the Moodle source code (see [[Git for Administrators]]). See the following procedure how to upgrade your site within several seconds to the most recent version while preserving your eventual local customizations tracked in git repository:<br />
<br />
$ cd /var/www/sites/moodle/htdocs/<br />
$ sudo -u apache /usr/bin/php admin/cli/maintenance.php --enable<br />
$ git pull<br />
$ sudo -u apache /usr/bin/php admin/cli/upgrade.php<br />
$ sudo -u apache /usr/bin/php admin/cli/maintenance.php --disable<br />
<br />
== Installation ==<br />
<br />
There are two modes of installing Moodle from the command line. In interactive mode, the install script asks you for all data needed to properly set up new Moodle site. In non-interactive mode, you must provide all required data as the script parameters and then the new site is installed silently. The parameters can be passed in the interactive mode, too. The provided values are then used as the default values during the interactive session.<br />
<br />
$ sudo -u apache /usr/bin/php admin/cli/install.php --lang=cs<br />
<br />
== Maintenance mode ==<br />
<br />
To switch your site into the maintenance mode via CLI, you can use<br />
<br />
$ sudo -u apache /usr/bin/php admin/cli/maintenance.php --enable<br />
<br />
To turn maintenance mode off, just execute the same script with --disable parameter.<br />
<br />
== Offline mode ==<br />
<br />
In some situations, you may want to switch your Moodle site into offline mode so that it is not accessible via the web but you can not stop the web server completely (typically because there are other web pages and applications running there). If a file called <code>climaintenance.html</code> exists in the root folder of moodledata directory, Moodle will automatically display the contents of that file instead of any other page.<br />
<br />
$ cd /var/www/sites/moodle/moodledata/<br />
$ echo '&lt;h1&gt;Sorry, maintenance in progress&lt;/h1&gt;' &gt; climaintenance.html<br />
<br />
You can prepare a nice formatted HTML page to inform your users about the server being down and keep in the moodledata directory under a name like <code>climaintenance.off</code> and rename it to the <code>climaintenance.html</code> if needed.<br />
<br />
== Custom site defaults ==<br />
<br />
During the install and upgrade via CLI, Moodle sets the administration variables to the default values. You can use different defaults. See MDL-17850 for details. Shortly, all you need to do is to add a file <code>local/defaults.php</code> into your Moodle installation. The format of the file is like<br />
<br />
<code php><br />
<?php<br />
$defaults['pluginname']['settingname'] = 'settingvalue'; // for plugins<br />
$defaults['moodle']['settingname'] = 'settingvalue'; // for core settings<br />
</code><br />
<br />
These defaults are used during install, upgrade and are also displayed as defaults on Site administration pages.<br />
<br />
== Reset user password ==<br />
<br />
If you happen to forget your admin password (or you want to set a password for any other user on the site), you can use reset_password.php script. The script sets the correctly salted password for the given user.<br />
<br />
$ sudo -u apache /usr/bin/php admin/cli/reset_password.php<br />
<br />
== MySQL storage engine conversion ==<br />
<br />
If you run your Moodle site with MySQL database backend and use the default MyISAM as the storage engine for your tables, you may want to convert them to use some more reliable engine like InnoDB (actually, you should want to switch to PostgreSQL ;-) anyway).<br />
<br />
$ sudo -u apache /usr/bin/php admin/cli/mysql_engine.php --engine=InnoDB<br />
<br />
==Converting InnoDB tables to Barracuda==<br />
<br />
Some users are getting the following MySQL error during course restore:<br />
<br />
Row size too large (>8126). Changing some columns to TEXT or BLOB or using ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED may help.<br />
<br />
The error is due to the fact that the MySQL default InnoDB file format (Antelope) cannot handle more than 10 text columns. Thus you are recommended to change the InnoDB file format to Barracuda using the following command:<br />
<br />
php admin/cli/mysql_compressed_rows.php<br />
<br />
For further information on InnoDB file formats see:<br />
<br />
* http://dev.mysql.com/doc/innodb/1.1/en/glossary.html#glos_antelope<br />
* http://dev.mysql.com/doc/innodb/1.1/en/glossary.html#glos_barracuda<br />
<br />
== Running cron via command line ==<br />
<br />
In versions 1.x, you could execute admin/cron.php either from command line or via the web. Since Moodle 2.0, only admin/cli/cron.php script can be run via command line.<br />
<br />
== Scheduled tasks ==<br />
<br />
Scheduled tasks are automatically run by the cron script, but the specific tasks which run on each cron iteration are determined by the scheduled tasks configuration. It is possible to override the scheduled tasks configuration and run a single scheduled task immediately using the admin/tool/task/cli/schedule_task.php script. <br />
<br />
This script accepts the following arguments:<br />
<br />
--list - list all the known scheduled tasks. The tasks are listed by the class name used to run the task. This class name is required as the argument to the next option in order to run a specific task immediately.<br />
<br />
--execute=<task> - Runs a single scheduled task immediately - regardless of scheduling settings. This will even run disabled tasks. Tasks will still use locking to prevent concurrent execution of the same task - even on clusters. The format of the <task> argument must be the same as returned by the --list option above.<br />
<br />
'''Note:''' You must escape the "\" with an extra \ when using the --execute command. Take the following for example:<br />
<br />
<pre>php schedule_task.php --list</pre><br />
<br />
will return something like:<br />
<br />
<pre><br />
== List of scheduled tasks (http://yourserver.com/moodle) ==<br />
\enrol_imsenterprise\task\cron_task 10 * * * * * ASAP<br />
\logstore_legacy\task\cleanup_task * 5 * * * * ASAP<br />
\logstore_standard\task\cleanup_task * 4 * * * * Wednesday, November 12, 2014, 4:35 AM<br />
\mod_forum\task\cron_task * * * * * * ASAP<br />
\core\task\automated_backup_task 50 * * * * * ASAP<br />
<br />
...<br />
</pre><br />
<br />
To run the first task in that list, you would execute<br />
<br />
<pre>php schedule_task.php --execute=\\enrol_imsenterprise\\task\\cron_task</pre><br />
<br />
==Database transfer==<br />
<br />
A command line script for [[Database transfer]] may be found in ''admin/tool/dbtransfer/cli/migrate.php''.<br />
<br />
==Purge caches==<br />
<br />
You can purge caches using this script:<br />
<br />
php admin/cli/purge_caches.php<br />
<br />
==Fix course / module sequences==<br />
<br />
In rare cases (such as after upgrading from a very old version of Moodle), the course / section / module sequence data can be out of sync. This can cause various problems for affected courses, such as sections not appearing, backups failing, pages not displaying etc. There is a specific check to check for errors caused by this problem, and to fix the data in the database if they are found. To run this script please use the command below:<br />
<br />
php admin/cli/fix_course_sequence.php -c=* --fix<br />
<br />
This will check every course in Moodle and report which ones had errors and were fixed.<br />
<br />
==Search and replace text==<br />
<br />
This script can be used to search and replace text throughout the whole database. Use carefully and backup first always. More info in [[Search and replace tool]].<br />
<br />
php admin/tool/replace/cli/replace.php --search=//oldsitehost --replace=//newsitehost<br />
<br />
==See also==<br />
<br />
* MDL-35736 - Manage plugins via command line<br />
* MDL-36237 - Resort course list via CLI<br />
<br />
<br />
[[fr:Administration en ligne de commande]]<br />
[[de:Administration über Kommandozeile]]<br />
[[ja:コマンドライン経由の管理]]<br />
[[es:Administración por línea de comando]]</div>Dmonllao