Better Office Integrations 3.3

Revision as of 01:55, 15 May 2017 by Damyon Wiese (talk | contribs) (REQ 1)

Jump to: navigation, search
Better Office Integrations
Project state Development in progress
Tracker issue https://tracker.moodle.org/browse/MDL-57794
Discussion https://moodle.org/mod/forum/discuss.php?d=346085
Assignee HQ Projects Team


Project - Better Office Integrations - Moodle 3.3

Overview

This document lists the minimum set of standard features that should be common to a premium standard office integration with Moodle. This document is not focussed on any particular office system. It’s likely that G-Suite and Office365 are the initial targets, but other systems could also be implemented in the same way in future.


Requirements

Site config

REQ 1

Users affected: Admins

The problem: The integration involves many plugins and there needs to be a single place to disable/enable it.

The proposed solution: Single admin setting to completely enable / disable all plugins for this integration.

Criteria for success:

  1. Admin can enable / disable the integration with a single setting.

Status: Completed for Moodle 3.3

Authentication

REQ 2

Users affected: All

The problem: If my Moodle site allows authentication with oauth - I should have all the possible OAuth providers displayed in priority order on the login page.

The proposed solution: Extend auth plugins API to support “oauth” authentication. Display the list of enabled OAuth plugins on the login page (in priority order).

Criteria for success:

  1. I see a list of “Login with XXX” options on the Moodle login page in priority order with icons and colors from the OAuth provider.
  2. Choosing an OAuth provider from the login page should open the external OAuth page in an popup window. When the OAuth is completed the popup window should close and the login page should reload and now be authenticated.
  3. If there is no matching Moodle account for the OAuth account - a new Moodle account should be created automatically and the OAuth account should be added to the list of connected OAuth accounts for this Moodle account.
  4. If manual authentication is disabled I should not see a username and password field on the login page. (Update - manual authentication cannot be disabled).

Status: Completed for Moodle 3.3

REQ 3

Users affected: Administrators

The problem: I want to allow users to login with their office accounts - but only if they come from the correct institution.

The proposed solution: Admin configuration settings to control the list of accounts allowed to login via OAuth (e.g. regex on username)

Criteria for success:

  1. Users can login to Moodle using an office suite account - only if it belongs to the correct institution.

Status: Completed for Moodle 3.3

REQ 4

Users affected: All

The problem: I do not want multiple different accounts to connect to different applications provided by my institution.

The proposed solution: Moodle authentication plugin supporting single sign on (OAuth).

Criteria for success:

  1. I can login to Moodle using my office account.
  2. If my account does not exist in Moodle it is automatically created.
  3. I have only one place to manage my account password.
  4. If I am logged into Moodle I can access my office applications without re-authenticating.
  5. If I am logged into my office applications I can access Moodle without re-authenticating.

Status: Completed for Moodle 3.3

REQ 5

Users affected: All

The problem: I want to access documents in my personal office account, while logged into Moodle with my university login

The proposed solution: Core api in Moodle for managing a list of oauth connected accounts and current sessions.

Criteria for success:

  1. I can login to Moodle using my university account (any existing auth method)
  2. When I access a resource from the office suite, I am prompted to login with a choice of my linked OAuth accounts (unless I already have a valid session).
  3. After logging in I can browse and search from the files connected to the chosen OAuth account.
  4. I can choose at any time to switch to a different connected OAuth account without ending my Moodle session.
  5. From the front page, I can login directly using an office account

Status: Completed for Moodle 3.3

REQ 6

Users affected: All

The problem: I want manage the list of OAuth accounts connected to my Moodle account

The proposed solution: Preferences page for managing the list of connected OAuth accounts

Criteria for success:

  1. I can see on my preferences page a list of connected OAuth accounts and can add and remove them.
  2. I can add a new connected OAuth account for any of the enabled OAuth Authentication plugins.
  3. I can login to Moodle using any of my connected accounts from the login page.
  4. I can prevent Moodle from creating accounts when someone logs with an oauth account and no matching user is found

Status: Completed for Moodle 3.3

Calendar

REQ 7

Users affected: Students, teachers

The problem: Calendar information is difficult to access if it is not aggregated in a single place.

The proposed solution: Automatically export the information from the Moodle calendar to the external calendar.

The criteria for success:

  1. My external calendar application automatically reflects the information on my Moodle calendar.
  2. I should be able to quickly hide / show all the Moodle events in my external calendar.
  3. If I cannot easily hide the events in my external calendar, I should be able to easily prevent Moodle from publishing events to it.

Update - This requirement is not possible to fulfill with the existing apis from Google and Microsoft.

Files

REQ 8

Users affected: Students, teachers

The problem: Users should be able to create files in the office applications and then use the files in Moodle.

The proposed solution: Add a repository plugin to Moodle that supports browsing of files in the office application.

The criteria for success:

  1. Any user can upload any office document they have access to wherever Moodle allows a file to be uploaded.
  2. They should be able to search or browse the documents in the office system.
  3. Once a document has been linked to Moodle. The access restrictions should rely on Moodles systems for capabilities roles etc (If I can see the link in Moodle I should be able to access the document).

Status: Completed for Moodle 3.3

REQ 9

Users affected: Students, teachers

The problem: I should be able to upload and create documents in the office suite without leaving Moodle.

The proposed solution: Add a direct link to the management interface for Google drive / Microsoft OneDrive.

The criteria for success:

  1. When browsing an office suite repository, I can easily open the native browsing interface in a new window.

Status: Completed for Moodle 3.3

REQ 10

Users affected: Students, teachers

The problem: Any file linked or added to Moodle should be available for checking via a plagiarism plugin

The proposed solution: Verify that files linked to an office suite are supported by the plagiarism API

The criteria for success:

  1. Verify any user file linked or copied from an office suite is available to the plagiarism API.

Status: The File API allows downloading of files, but no plagiarism plugins exist in core - so no changes were made.

REQ 11

Users affected: Developers

The problem: It is difficult to test and maintain the plagiarism API because we have no core implementations and third party implementations require accounts / setup.

The proposed solution: Create a dummy plagiarism plugin (google search?) with no installation requirements.

The criteria for success:

  1. Developers can install a plagiarism plugin to easily verify the correct functioning of the plagiarism API.

Status: Not complete.

REQ 12

Users affected: Administrators

The problem: Institutions can have policies about where to upload and store files used in Moodle

The proposed solution: Create admin settings for each repository plugin to control the default (link or copy) and optionally disable either option.

The criteria for success:

  1. The administrator configured default (link or copy) is automatically chosen when users are browsing files from a repository.
  2. If link or copy are disabled by the administrator, they are not available as options for users when browsing files from a repository.

Status: Completed for Moodle 3.3

REQ 13

Users affected: Students, teachers

The problem: Users should be able to easily update a file that has been linked to Moodle

The proposed solution: Allow repository plugins to generate an “edit link” for a document that is displayed in the UI in addition to the link to view the file.

The criteria for success:

  1. Any user who can update a file in Moodle that was linked from an office system has a direct link in Moodle to edit the file in the office suite.
  2. After saving changes to the file in the office suite - anyone who views the file in Moodle should see the latest version of the file.

Status: Completed for Moodle 3.3 (although there is a single link to the file).

REQ 14

Users affected: Students, teachers

The problem: Files available in Moodle should be available offline, and on the Mobile app.

The proposed solution: Verify any read or edit links to files work correctly in the Mobile app. Add download for offline access links to any document where it is available.

The criteria for success:

  1. There is a way to download linked files from Moodle for viewing offline.
  2. Edit or view links work correctly when used in the Mobile app.

Status: Completed for Moodle 3.3

REQ 15

Users affected: Students, teachers

The problem: Backing up a Moodle course should backup all the externally linked documents.

The proposed solution: Verify the behaviour of backup / restore.

The criteria for success:

  1. Backing up a course with external files should include only references to the files.
  2. Restoring a course with linked files to the same site, should restore any files that were previously external links as external links to the same original files.
  3. Restoring a course with linked files to a different site not restore the files, but should show warnings instead.

Status: Completed for Moodle 3.3

Assignments

REQ 16

Users affected: Students, teachers

The problem: It should be possible for activities to control access to a file that has been linked.

The proposed solution: Extend the repository API to allow creating folders, copies of files to a new owner, and changing the permissions on a file/folder. Extend the assignment activity to implement a workflow for files used as assignment submissions.

The criteria for success:

  1. A student can upload any file they have access to as an assignment submission.
  2. A student can override their assignment submission with a later version
  3. All members of a group should be able to collaborate on a group submission before adding it as their assignment submission.
  4. Teachers should have read / write access to the submitted files.
  5. After the files have added to Moodle students should only be able to “read” the file (but they can link a new version of the file as long as they have access to do so).
  6. It should be possible for any Moodle plugin to replicate this same workflow via the repository API.

Status: Completed for Moodle 3.3

Archiving

REQ 17

Users affected: Students, teachers

The problem: If a teacher links to a file in their personal office suite file area, and then the teacher account is deactivated, deleted or disconnected from the office suite - users in the course must still be able to read and write to the file.

The proposed solution: Verify that access to a file is not lost if the office account no longer exists. Store a copy of the file in Moodle to be used if the original file cannot be accessed.

The criteria for success:

  1. If a linked file in Moodle is deleted from the office suite - it should still be available to download in Moodle.
  2. If the account that linked to a file in an office suite is removed - it should still be possible to download the file in Moodle and ideally it should still be possible to edit the file.

Status: Completed for Moodle 3.3

REQ 18

Users affected: Administrators

The problem: If files owned by an “Institution” account in the office suite are not organised - it will be impossible to archive no-longer used files at the end of a teaching period.

The proposed solution: All files transferred to an institution account as part of a custom workflow should be organised by course / activity so they can be manually archived as required.

The criteria for success:

  1. Files used in a custom workflow (e.g. by assignment) should stored in an institutional account, and organised by category / course / activity (reflect the context structure in Moodle).

Status: Completed for Moodle 3.3

User Experience

REQ 19

Users affected: Students, teachers

The problem: When connected to an office suite - it should be possible to upload or create a file in the office suite with a minimum number of steps.

The proposed solution: When using the file picker, there should be a direct link to the native browsing interface which provides the best experience for uploading and creating new files.

The criteria for success:

  1. Each repository should have a direct link to the native browsing interface.

Status: This requirement was not addressed for Moodle 3.3. It should really be done as a bigger project reviewing the UX of the file picker entirely.

Technical Specifications

This section of the document describes the changes that need to be made in Moodle in order to meet the success criteria of each of the requirements for this project. These specifications describe the design of each of the Office 365 and Google Apps integrations and so each section needs to be implemented twice - once for each integration.

Core API for managing Authorized OAuth Applications

There should be a new preference page and API to manage the list of a users authorized oauth applications.

This requires a new DB tables to store the system wide OAuth clients and the current users active oauth sessions should be stored in the current PHP session.

mdl_oauth2_clients

  1. clientid - varchar(100) - NOT NULL
  2. secret - varchar(100) - NOT NULL
  3. refreshtoken - varchar(255) - NULL
  4. systemuser - varchar(255) - NULL
  5. apirooturl - varchar(255) - NOT NULL
  6. apirequesttokenurl - varchar(255) - NOT NULL
  7. apiaccesstokenurl - varchar(255) - NOT NULL

The apirooturl + apirequesttokenurl + apiaccesstokenurl form a unique key. It is possible to retrieve the stored clientid and secret using the 3 api url parameters.

The refresh token here is used to get an access token for the system account.

The systemuser is the unique identifier required to access resources using a system account rather than the current users account.

Notes (after prototyping).

Open ID Connect is the SSO standard built on top of OAuth 2.0. Google supports it well and Microsoft claim to - but their implementation is not functional/usable.

Problems with Microsoft Open ID Connect implementation (their side not ours):

  1. It's userinfo endpoint is missing email and picture claims.
  2. The discovery document points to the wrong api version (1.0 instead of 2.0)
  3. The tokens from the Open ID Connect token endpoint are not usable with the graph API (because of b)
  4. The tokens from the alternative OAuth 2.0 endpoints are not compatible with the OpenID Connect endpoints (userinfo).
  5. Not necessarily a problem but the sup claim from Microsoft is a massive unreadable string which is not nice for a username (but is guaranteed unique).

Extend Auth Plugin API

The existing Auth Plugin API should allow plugins to register themselves as "OAuth 2 " compatible. This will allow them to be listed on the login page in priority order with icons and coloured buttons. The order of the buttons should be determined by the order of the auth plugin.

Auth plugin

Implement a new authentication plugin that is "OAuth 2" compatible and is preconfigured to talk to the office suite.

Provide admin settings to limit the valid domains / institutions that can authenticate with this plugin.

Add a setting to allow self registration with this plugin so that accounts can be automatically created if the authentication is successful and no matching account exists.

When an account is connected with this auth plugin - the Moodle calendar rss url should be published to the Office calendar automatically.

There should be a setting to disable the automatic publishing of calendars.

Admin tool to manage integration settings

An admin tool needs to be created which contains the required information in order to access the office suite API using OAuth as well as a valid access token for a institutional administrator account.

This plugin needs a global on/off switch which enabled or disables the entire office suite integration (config value stored in mdl_config_plugins).

This plugin should have settings to create an entry in the mdl_oauth2_clients table.

The workflow for setting up the plugin would be to enter the client id and secret, then follow a link in the admin tool to authenticate the institutional account via OAuth 2. The refresh token for the institution account will be stored in the database - and used whenever we need to access the API as the "institution" rather than the current user.

Repository API

The repository api needs to be extended so that repositories can support advanced workflows for files.

Repository plugins should be able to register that they are OAuth compatible. This will enable an additional configuration page for the plugin which allows setting up the system account and authorizing a refresh token for this system user. The refresh token will then be used by the repository plugin to generate an access token when it needs to perform an operation as the system user.

The additional functionality required is:

get_edit_link() - retrieve a link to edit this file in an external application. If supported will show an additional link or menu entry when the file is listed to "edit this file". Moodle will only generate and display this link to users who have access to edit the file in Moodle. The external repository should allow editing from any user who follows this link regardless of the permissions in the external repository.

get_offline_link() - for external files (links), retrieve a link to download this file in a format that is suitable for viewing offline - e.g. PDF. Moodle will only generate and display this link to users who have access to read the file in Moodle. The external repository should allow viewing from any user who follows this link regardless of the permissions in the external repository.

copy_to_system_account() - copy a file from the users file area in the external repository - to new file in a system owned file area in the external repository organised by category, course and activity. This is used as part of an activity workflow when the file switches from being editable by the original author - to being editable by a different group of users (teachers most likely).

Repository Plugin

Implement a new plugin for this office suite that allows browsing and searching the users currently authorized (OAuth) file area. If there is no valid OAuth session for this OAuth client API the OAuth login screen should be presented to create a new session.

This plugin should also support all of the new functionality for repository plugins listed in Repository API above.

This plugin should also support uploading and creating new files in the repository while browsing / searching.

This plugin should contain a setting to "download for archive" which should download an offline copy of each file - even if was created as a link and store it in Moodle. If enabled the "offline" version of this file will be served from Moodle always and included in backup/restore. Offline versions of files should be refreshed by a cron task and an adhoc task should be created to check for a newer version when a file is accessed.

This plugin should have a setting to say if the default for files added to Moodle is "link" or "copy". It should be possible to disable "link" or "copy" (but not both).

Assignment Improvements

The assignment should check if the repository plugin supports the advanced workflow for submissions files. If it does - it should copy the submission file to a system account at the point that it is "submitted".

Improvements to the File Picker

If there are any enabled repositories that report they support uploading of files - a new button should be added to the file manager "Upload file" which will open the file picker with the first "upload" repository already selected.

Plagiarism API

Extend the plagiarism API to check if a repository supports "offline" links. If it does - the offline version of files should be downloaded and used by the plagiarism API as normal.

Plagiarism Plugin

Implement a dummy plagiarism plugin to make testing easier. It should support all features of the plagiarism API but does not have to return real data.