開発:ファイルAPI

Moodle 2.0 このページは、Moodle 2.0でのファイルストレージおよびアクセスの実装に関する、現在考えていることに関する概要ページです。作成中の仕様書です!

作成中です - Mitsuhiro Yoshida 2009年1月6日 (火) 09:16 (CST)

このページは、誰でも編集することができますので、誰でも間違いを訂正したり、このドキュメントを進化させることができます。あなたに質問、報告する問題または主要な変更に関する提案がある場合、ページコメント欄に追加するか、リポジトリフォーラム (英語)でディスカッションを開始してください。開発を開始する前、私たちは、それらのすべてのご意見をメインの仕様に取り込むよう努力します。

目的

1. ファイルを直接Moodleに追加できるようにする (現在、私たちが追加しているように)。
2. ファイルがどこから来たのか記憶する。
3. ケイパビリティおよび他のローカルルールを使用して、モジュールにファイルアクセスのコントロールを与える。
4. Moodle内のすべてのファイル処理に関して、一貫してシンプルにアプローチする。

オーバービュー

ファイルAPIは、Moodleコードが次の目的で使用するインターフェースのコアです:

1. Moodle内にファイルを保存する
2. Moodleユーザにファイルを表示する

ファイルAPIは「ユーザ」ファイルのみ提供します。また、ファイルAPIは、次のディレクトリのような、Moodleのdatarootディレクトリに作成されたローカルファイルおよびキャッシュを提供しません: temp、lang、cache、environment、filter、rss、search、sessions, upgradelogs等

APIは、いくつかの独立したパーツに分かれます:

1. ファイル提供API
   1. file.php
   2. pluginfile.php
   3. userfile.php
   4. draftfile.php
   5. rssfile.php
2. ファイル保存API
   1. 任意のアクセスコントロール
   2. 任意のリポジトリsync
3. ファイル管理API
   1. ファイル閲覧
   2. ファイルリンク (エディタ統合)
   3. リポジトリからのアップロード

ファイル提供API

ファイルの提供に対応します - ブラウザがファイルをリクエストした場合、そのファイルをMoodleが返します。私たちには、3つのメインファイルがあります。サーバ上でスラッシュ (例 file.php/some/thing/xxx.jpg) を設定することが重要です。相対リンクを使用するコンテンツは、スラッシュなしではアクセスできなくなります (scorm、アップロードhtmlページ等)。

file.php

コースファイルを提供します。

基本的なファイルを扱います。理想的には、コースセクションからリンクされたイメージおよびファイルのみ扱われるべきです。XSS保護は必要としません - 私たちはjavaスクリプト、sw (shockwave) 等の使用を想定しています。「セキュア」にする方法はありません。私たちがほとんどのファイルをモジュール内に移動する場合、アクセスコントロールは重要な意味を持ちません。

既存のコースコンテンツに対する下位互換性に関して、ファイル名およびパラメータ構造は重要です。

/file.php/courseid/dir/dir/filename.ext

内部的にファイルは、array('contextid'=>$coursecontextid, 'filearea'=>'coursefiles', 'itemid'=>0)に保存されます。

pluginfile.php

(別名 modfile.php) モジュール、ブロック、問題ファイルを送信します。

• モジュールは、アクセスコントロールを決定します。
• 任意のXSS保護 - 学生が送信したファイルは、ノーマルヘッダと共に提供されるべきではありません。私たちは、代わりにダウンロードさせる必要があります。理想的には、信頼できないファイルに対して、第2のwwwrootを使用した方が良いでしょう。
• 選択したエリアの内部リンクのみサポートされます - あなたは要約エリアのみイメージをリンクすることができますが、提出課題にはリンクすることができません。

モジュールでHTML編集が許可されている場合、絶対リンクは書き換えられる必要があります。リンクは、相対リンクとして内部に保存されます。編集または表示する前、str_replace()を使用して、内部リンクは絶対リンクに変換されます (@@thipluginlink/summary@@/image.jpg --> /pluginfile.php/assignmentcontextid/intro/image.jpg)。保存される前、内部リンクに戻されます。

明確な (distinct) ファイルエリアは、情報を追加するため、どこかで定義されたプラグインによってサポートされますか? 例えば、以下のように宣言されると面白いと思います:

• assignment_summary:
  • relpath='intro'
  • userdata=false
  • anotherproperty=anothervalue
• assignment_submission:
  • relpath='submission/@@USERID@@'
  • userdata=false
  • anotherproperty=anothervalue
• など ...

それから、エディタが「assignment_summary」エリア名を「受信」した場合、何を表示するのか等、知っているとしたら? また、バックアップおよびリストアにおいて、ファイルエリアが処理されるか否か (userdata=fals) 判断するため、その情報が有用だとしたら。または、リンクの再構築 (str_replace() above) に対して、その情報が有用だとしたら。そして、自由にコーディング (prone to errors 間違いがち) する代わりに、モジュールによって良く定義されたファイルエリアのリストがあるとしたら。Eloy Lafuente (stronk7) 16:35, 28 June 2008 (CDT)

このような機能は、ファイル管理APIの一部にありますが、これをファイル保存のコードにハードコードすると柔軟性が失われます、私の考えでは。Skodak

はい、はい。ストレージは、ファイルをget/putする以外、何も知ってませんね。確かに、これは管理の一部ですね。Eloy Lafuente (stronk7) 11:21, 29 June 2008 (CDT)

/pluginfile.php/contextid/areaname/arbitrary/params/or/dirs/filename.ext

pluginfile.phpは、コンテクストテーブルよりプラグインタイプを検出し、基本情報 (状況に応じて、$courseまたは$cm) を取得します。また、アクセスをコントロールして最終的にファイルをユーザに送信するプラグイン関数 (または後のメソッド) をコールます。areanameは、タイプ別にファイルを分離して、コンテクストをいくつかのサブツリーに分けます - 例えば、summaryファイル (モジュールのイントロダクションで使用されるイメージ)、投稿の添付ファイル等。

課題例

/pluginfile.php/assignmentcontextid/intro/someimage.jpg
/pluginfile.php/assignmentcontextid/submission/submissionid/attachmentname.ext
/pluginfile.php/assignmentcontextid/extra/allsubmissionfiles.zip

Uhm... all those files together? What's going to differentiate the "submission" path in the example above from the "summary" path? Is it supposed that the editor, or the filemanager won't allow , for example to pick-up one file from the "submission" area to be used in the summary of one assignment and only the "summary" area will be showed? That means multiple file managers by context and it's against the clean "one file manager per context" agreed below Eloy Lafuente (stronk7) 21:28, 26 June 2008 (CDT)

Yes Eloy, the different areas (summary, submission) etc. have different uses, different access control. There are two types of file manager - the two pane file manager which lists all contexts+areas user may access, and minimalistic manager in html editor which shows only subset of areas from current plugin (because you can not link anything else).

SCORM例

/pluginfile.php/scormcontextid/intro/someimage.jpg
/pluginfile.php/scormcontextid/content/revisionnumber/dir/somescormfile.js

The revision counter is incremented when any file changes in order to prevent caching problems. The lifetime should be adjustable in module settings.

小テスト例

pluginfile.php/quizcontextid/intro/niceimage.jpg
pluginfile.php/quizcontextid/report/type/export.ods

問題例

pluginfile.php/SYSCONTEXTID/question/questionid/file.jpg

ブログ例

Blog entries or notes in general do not have context id (because they live in system context, SYSCONTEXTID below is the id of system context). The note attachments are always served with XSS protection on, ideally we should use separate wwwroot for this. Access control can be hardcoded.

/pluginfile.php/SYSCONTEXTID/blog/blogentryid/attachmentname.ext

Internally stored in array('contextid'=>SYSCONTEXTID, 'filearea'=>'blog', 'itemid'=>$blogentryid)

バックアップ例

It would be nice to have some special protection of backup files - new capabilities for backup file download, upload. Backups contain a lot of personal info, we could block restoring of backups from other sites too.

/pluginfile.php/coursecontextid/backup/backupfile.zip

Internally stored in array('contextid'=>$coursecontextid, 'filearea'=>'backup', 'itemid'=>0)

userfile.php

Personal file storage, intended as an online storage of work in progress like assignments before the submission.

• read/write own files only for now
• option to share with others later
• personal "websites" will not be supported (security)

/userfile.php/userid/dir/dir/filename.ext

rssfile.php

Replaces rss/file.php which is kept only for backwards compatibility. RSS files should not require sessions/cookies, URLs should contain some sort of security token/key. Internally the files may be stored in database or together with other files. Performance improvements - we should support both Etag (cool) and Last-Modified (more used), when we receive If-None-Match/If-Modified-Since => 304

/rssfile.php/contextid/any/parameters/module/wants/rss.xml
/rssfile.php/SYSCONTEXTID/blog/userid/rss.xml

Again modules and plugins decide what gets sent to user.

一時ファイル

Temporary files are usually used during the lifetime of one script only. uses:

• exports
• imports
• zipping/unzipping
• processing by executable files (latex, mimetex)

Ideally these files should never use utf-8 (which is a major problem for zipping at the moment). Proposed new sha1 based file storage is not suitable both for performance and technical reasons.

レガシーのファイル保存および提供

Going to use good-old separate directories in $CFG->dataroot.

file serving and storage:

1. user avatars - user/pix.php
2. group avatars - user/pixgroup.php
3. tex, algebra - filter/tex/* and filter/algebra/*
4. rss cache (?full rss rewrite soon?) - backwards compatibility only rss/file.php

only storage:

1. sessions

ファイル保存API

Modules in general work only with local Moodle files. One of the major reason is performance when accessing external repository files. It will be possible to use repositories instead of file uploading and also to keep local files synced with external repository.

File contents are stored in moodledata/filepool indexed using SHA1 hashes instead of file names; file names, relative paths and other metadata will be stored in file(_xxx) database tables. This should be fully abstracted so that modules do not actually know where the files are located. When storing files the content is sent as string or file handle, when reading content it is returned as file handle.

ファイルテーブル

This table contains one entry for every file. Enough information is kept here so that the file can be fully identified and retrieved again if necessary.

note: plural used because file is a reserved word

Field	Type	Default	Info
id	int(10)		autoincrementing
contenthash	varchar(40)		The sha1 hash of content.
pathnamehash	varchar(40)		The sha1 hash of contextid+filearea+itemid+filepath+filename - prevents file duplicates and allows fast lookup
contextid	int(10)		The context id defined in context table - identifies the instance of plugin owning the file.
filearea	varchar(50)		Like "submissions", "intro" and "content" (images and swf linked from summaries), etc.; "blogs" and "userfiles" are special case that live at the system context.
itemid	int(10)		Some plugin specific item id (eg. forum post, blog entry or assignment submission or user id for user files)
filepath	text		relative path to file from module content root, useful in Scorm and Resource mod - most of the mods do not need this
filename	varchar(255)		The full Unicode name of this file (case sensitive)
filesize	int(10)		size of file - bytes
mimetype	varchar(100)	NULL	type of file
userid	int(10)	NULL	Optional - general user id field - meaning depending on plugin
timecreated	int(10)		The time this file was created
timemodified	int(10)		The last time the file was modified

non-unique index on "contextid, filearea, itemid" and "contenthash"; unique index on "pathnamehash"

Plugin type is not specified because it is derived from contextid, items like blog that do not have own context will use own filearea usually from systemcontextid.

Perhpas we could also hash filepath and filename and index by them, to save some text limitations in the DB side (length limits of indexes, not indexable, complex retrieval...). Eloy Lafuente (stronk7) 11:54, 29 June 2008 (CDT)

Also, perhaps we should store finally the plugin type there to save some queries per request, using it to drive to the correct file handling of each plugin. Eloy Lafuente (stronk7) 18:54, 29 June 2008 (CDT)

files_cleanup table

This table contains candidates for deletion from the file pool. Files are not deleted immediately, cron uses the files_cleanup table, verifies the file is not used any more and deletes it from pool. Reasons for cron clean-up are performance and prevention of collision - there could be a problem with concurrent uploads and deletes, we will probably need to add some table-based locking during the clean-up.

We might add an extra script that does deep validation of pool area - report missing files, report orphaned files, content not matching the sha1 filename, etc. - this would very very time consuming.

Field	Type	Default	Info
id	int(10)		autoincrementing
contenthash	varchar(40)

files_metadata table

This table contains extra metadata about files. Repositories could provide this, or it could be manually edited in the local copy.

Field	Type	Default	Info
id	int(10)		autoincrementing
fileid	int(10)		Id of file.
name	varchar(255)		The name of extra metadata
value	text		Value

files_acl table

This table describes optional ACL for file. This is not required in majority of cases, modules usually hardcode the file access logic, course files should not be used much any more.

Field	Type	Default	Info
id	int(10)		autoincrementing
fileid	int(10)		The file we are defining access for
contextid	int(10)		The context where this file is being published
capability	text		The capability that is required to see this file.

acl notes

• this is missing some concept similar to user/group/others, for example in case of user files typical user can not assign permissions or view them - this becomes useless there
• it is more important to synchronise the availability of file link and the file itself - having link pointing to inaccessible file or file which is accessible when not wanted are both problems
• browser/proxy caching works against us here - "secret" files should not be cached

files_sync table

This table contains information on how to synchronise data with repositories. Data would be synchronised from cron.php or on demand from file manager. The sync would be one way only (repository-->local file).

Field	Type	Default	Info
id	int(10)		autoincrementing
fileid	int(10)		Id of file.
repositoryid	int(10)		The repository instance this is associated with, see Development:Repository_API
updates	int(10)		Specifies the update schedule (0 = none, 1 = on demand, other = some period in seconds)
repositorypath	text		The full path to the original file on the repository
timeimportfirst	int(10)		The first time this file was imported into Moodle
timeimportlast	int(10)		The most recent time that this file was imported into Moodle

File content storage

Originally the file storage hierarchy contained a lot of metadata including userids, entry ids, filenames, etc. The file content will now be stored separately from file metadata. It must supports utf8 on all platforms.

File storing:

1. calculate SHA1 hash of content
2. check if file with SHA1 name exists, if not add the file to file pool
3. remove SHA1 from list of deleted files if found there
4. store file in file table, use SHA1 as file pool identifier

File reading:

1. fetch file record from 'file' table - probably using file id or combination of contextid+instanceid
2. fetch content of file

File deleting:

1. delete record from file table, remember file SHA1
2. store the deleted SHA1 in deleted files table, do not remove the physical file yet
3. wait for cron cleanup script to actually delete the file named SHA1 (proper table locking needed to prevent race conditions when adding/deleting files)

File pool details

located in $CFG->dataroot/filepool/, all files can not be stored in one directory due to OS limitations, it uses 3 levels based on first three characters of sha1 hash. It is unlikely that there will be thousands of files with the same first 3 chars in sha1 hash of their content.

This type of storage saves a lot of disk space when storing multiple copies of the same large file. It can also help substantially when synchronising data with external repositories. Another benefit is we can detect inconsistencies in file content.

File read performance is similar to previous code, file write performance will be slower - due to hashing and extra database access.

dataroot 
   /filepool
      /00
      /01
      ...
      /23
        /00
        /01
        ...
        /1e
           /00
           /01
           ...
           /2d
              /231e2dc421be4fcd0172e5afceea3970e2f3d940
      ...
      /fe
      /ff

File management API

This section describes following:

1. file manager
2. integration with html editor
3. interactions with repos

File manager

Single pane file manager is hard to implement without drag & drop which is notoriously problematic in web based applications. I propose to implement a two pane commander-style file manager. Two pane manager allows you to easily copy/move files between two different contexts (ex: courses).

File manager must not interact directly with filesystem API, instead each module should return traversable tree of files and directories with both real and localised names (localised names are needed for dirs like backupdata).

Originally there was a single file tree for each course. We need to fully separate each module/block from the course files and there might be also independent file areas in modules (ex: module introduction, content files, submissions, post attachments). File area may be defined as a small tree where we can use relative paths. These file areas are hanging from the branches of the context tree (this needs a picture).

Integration with htmleditor

Html editor should be able to browse only relevant files - for example when editing resource introduction only images from the file area of that resource should be available; when editing html resource page only the content area images should be listed.

There are several problems here:

1. when adding new resource its context does not exist yet, we will have to create some table to handle temporary file storage for adding of new stuff, not easy but should be solvable - maybe we could abuse the course context id or store it temporarily in some special user file area
2. we can not use absolute address relinking for pluginfile.php links, instead we can use the absolute links only when editing and before storage convert them to something like @@thispluginfile@@/2112/112/image.jpg before storage. the local links would be converted to full absolute links before display or editing. Not all file areas will support this (ex: linking to assignment submission does not make sense because nobody else may access it anyway). This would allow us to implement image preview in html editor.

Html editor should contain simplified single pane file manager with basic operations only - select file area, browse file area, upload file/copy user file/use repo file, delete. The editor will communicate with modules and core through ajax call to some script specified by module embedding the editor. The callback script would use different logic to construct the tree of files than the File manager, it needs to know only about files that other ppl viewing the resulting html may access.

Interactions with repos

Repositories may serve as a replacement for file uploading. They may be also used to synchronise files between courses. The repo option should be available whenever there is a file upload field, sometimes with extra "keep synchronised" option (this would not make sense for stuff like assignment submissions).

Upgrade, migration and backwards compatibility

It is going to be a pain again like DML/DDL ;-)

Code backwards compatibility

0% backwards compatibility related to file storage. New objects will be mandatory to use. Old $CFG->dataroot/$courseid/ will be empty, $CFG->dataroot/blog/ too, etc.

Content backwards compatibility

Means existing courses should not loose images, flash, etc. Though some new features (like resource sharing - if implemented) may not work with existing data that still uses files from course files area.

There might be a breakage of links due to special characters stripping in uploaded files which will not match the links in uploaded html files any more. This should not be very common I hope.

Migration of content

• resources - move files to new resource content file area; can be done automatically for pdf, image resources; definitely not accurate for uploaded web pages
• questions - image file moved to new are, image tag appended to questions
• moddata files - the easiest part, just move to new storage
• coursefiles - there might be many outdated files :-( :-(
• rss feeds links in readers - will be broken, the new security related code would break it anyway

Moving files to files table and file pool

The migration process must be interruptable because it might take a very long time. The files would be moved from old location, the restarting would be straightforward. Proposed stages:

1. migration of all course files except moddata - finish marked by some $CFG->files_migrated=true; - this step breaks the old file manager and html editor integration
2. migration of blog attachments
3. migration of question files
4. migration of moddata files - each module is responsible to copy data from converted coursefiles or directly from moddata which is not converted automatically

Some ppl use symbolic links in coursefiles - we must make sure that those will be copied to new storage in both places, though they can not be linked any more - anybody wanting to have content synced will need to move the files to some repository and set up the sync again.

Talked about a double task here, when migrating course files to module areas:

1. Parse html files to detect all the dependencies and move them together.
2. Fallback in pluginfile.php so, if something isn't found in module filearea, search for it in course filearea, copying it and finally, serving it.

Also we talked about the possibility of add a new setting to resource in order to define if it should work against old coursefiles or new autocontained file areas. Migrated resources will point to old coursefiles while new ones will enforce autocontained file areas.

it seems that only resource files will be really complex (because allow arbitrary HTML inclusion). The rest (labels, intros... doesn't) and should be easier to parse.

Eloy Lafuente (stronk7) 19:00, 29 June 2008 (CDT)

Backup/restore changes

File handling in backups needs to be fully rewritten - list of files in xml + pool of sha1 named files with contents. This solves the utf-8 trouble here, yay!!

Quotas

File size will be stored in files table, we can use simple queries to find out how much space is used, however this may not be accurate because the sha1 hash based storage eliminates duplicate files.

If sha1 string is stored in the files table (non-unique), then it can be used to detect duplicates within the files table and only counting their size once. To avoid having to search this often, we could do it periodically and store the filesize with each file record (only the first of the duplicates gets a filesize, the others get 0). Nicolas Connault

• total course files - find out all contexts used in course, query files table with contextid IN ($listofcontexts)
• module files - find module context and calculate space per file area
• user files quota - inside the personal area only, counting all attachments in all mods might take a while

We could also divide the file size by number of instances that are using it, this might be considered more accurate in some scenarios.

Other

• antivirus scanning + upload manager rewrite/integration with forms lib
• zip compression and extraction

Major problems

List of hard to solve prolbems

zip support

Zip format is an old standard for compression of files, it was created long before the unicode and added support for it just recently. There are several ways used for encoding of non-ascii characters in path names, unfortunately it is not much standardised. Most windows packers use DOS encoding.

Client software:

• Windows built-in compression - bundled with Windows, non-standard DOS encoding only
• WinZip - shareware, unicode option (since v11.2)
• TotalCommander - shareware, single byte(DOS) encoding only
• 7-Zip - free, unicode or DOS encoding depending on characters used in file name (since v4.58beta)
• Info-ZIP - free, uses some weird charset conversions

PHP extraction:

• Info-ZIP binary execution - no unicode support at all, mangles charsets in filename (depends on OS, see docs), files must be copied to temp directory before compression and after extraction
• PclZip PHP library - reads single byte encoded names only, problems with random problems and higher memory usage
• Zip PHP extension - reads single byte encoded names only, 64bit operating system can not open/create archives with more than 500 files (depends on sum of lengths of all filenames and directories, to be fixed in PHP 5.3 and external PECL library, no PHP 5.2.x backport planned!), adding of files is limited by number of free file handles (around 1000 - depends on OS and other PHP code, workaround is to close and reopen archive)

Large file support: PHP running under 32bit operating systems does not support files >2GB (do not expect fix before PHP 6). This might be a potential problem for larger backups.

Tar Alternative:

• tar with gzip compression - easy to implement in PHP + zlib extension (PclTar, Tar from PEAR or custom code)
• no problem with unicode in *nix, Windows again expects DOS encoding :-(
• seems suitable for backup/restore - yay!

Roadmap:

1. add zip processing class that fully hides the underlying library
2. use single byte encoding "garbage in/garbage out" approach for encoding of files in zip archives; add new 'zipencoding' string into lang packs (ex: cp852 DOS charset for Czech locale) and use it during extraction, we might support true unicode later when PHP Zip extension does that

empty directories

Hmm, thinking a bit more about Justin's comment I realised there is no support for empty directories in this proposal. This will require either new table or some hack in files table - maybe we could add files with "." as name and just skip them when iterating directory content.

Note: solved with files named '.'; the dirs are automatically created when adding files, the area root dir is created when browsing.

file overwriting

Concept of file overwriting does not exist anymore here, the path+filename are not enforced to be unique - we can not make index because sloppy mssql does not allow indexes larger than 900 bytes :-( We will haev to emulate it somehow and deal with collisions if found.

Note: solved with unique index on pathnamehash field.

Some little comments to be considered (to avoid forgetting them)

• each context will have its own "file manager"
• separate "file manager context" files (FMF) and "internal context" (ICF) files (current modedit files, submissions, attachements...)
• /pluginfile.php/SYSCONTEXTID/{blog|question} and so... will have own FMF too? Or only ICF ?
• Way to copy between contexts
• Links = -1 for them
• Deletion strategy (locks, quarantine status...)
• include support for quotas per user, per course, etc
• upgrade process should be interruptable (like the unicode upgrade) so it can be stopped/restarted any time

Justin's thinking out loud

I'm actually working on implementing this along with extending an existing Alfresco integration to work together with the whole File / Repository system and I wanted to get some of my comments and thoughts in here for feedback. Go easy on me. =)

So far I've only got one that I'd like to solicit some feedback on (BTW, if this would be better suited to a forum discussion, let me know):

Not storing the full filepath with each entry in the file table

• For browsing a directory structure, determining things like child directories or a parent directory given a filepath requires a lot of extraneous coding in PHP. I think it might be better served to create a new file_directory table, storing only a directory name, and reference to a parent directory record. The benefits here are that we're storing a lot of duplicate text field values in the file table and browsing through the file picker for local files doesn't require a lot of PHP overhead to calculate links to parent / child directories.
• Given that file permissions are no longer calculated using structured file paths, using the complete, full, path to a given file would most likely never be needed.

• The repositorypath field in the repository_sync table still makes sense, though.

• The file_directory table:

Field	Type	Default	Info
id	int(10)		autoincrementing
parent	int(10)		ID of directory that this record is a child of.
directoryname	varchar(255)		The actual name of this directory.

skodak: filepath is stored in files table - its root is the corresponding filearea, the file manager will use the context tree to find all plugins/courses and ask them to return the list of areas with all those small branches inside it

関連情報

• 開発:ファイルAPIの利用
• 開発:リポジトリAPI
• 開発:ポートフォリオAPI
• MDL-14589 - File API Meta issue