Development talk:File API
Some quick questions to avoid forgetting them:
1) Will them be under the control of FileAPI (or, as they are now, fixed local storage)?
- dataroot/temp - dataroot/lang - dataroot/cache - dataroot/environment - dataroot/filter - dataroot/rss - dataroot/search - dataroot/sessions - dataroot/upgradelogs
Martin Dougiamas 03:20, 28 April 2008 (CDT) : I don't see these as being in the API - I've updated the spec.
2) Assuming we'll have a cool OOP FileAPI...
- a) Will it support different FileAPI classes (to be able to store in other systems) ? - b) Will it support multiple FileAPI classes working together (like the Repo) ?
Martin Dougiamas 03:20, 28 April 2008 (CDT) : Hmm, I suppose it makes sense to switch the backend from local file storage to specify something else (eg database storage) but multiple File storage places doesn't make sense to me, that is the Repository API and the Portfolio API.
3) I've annotated in red some things that have sounded strange in my first look.
Martin Dougiamas 03:20, 28 April 2008 (CDT) : Thanks, all fixed.
4) Are we going to have "directory records" in the implementation, or that is going to be handled exclusively by the "moodlepath" column?
Martin Dougiamas 03:20, 28 April 2008 (CDT) : Good point. I was thinking of moodlepath only but I wonder if directory records might be more efficient. New table, I guess.
5) One general question... are we going to "force" all modules to be "autocontained" ? How are we going to handle resources, for example (with all those css, links, images..). In general how are we going to handle multiple-file packages?
Martin Dougiamas 03:20, 28 April 2008 (CDT) : they'll be a set of files, probably in a "directory" specified by moodlepath of directory record. What problems do you see? Should we retain better knowledge of the original group of files?
That's all for now, ciao4niao :-) Eloy Lafuente (stronk7) 21:27, 5 April 2008 (CDT)
Making Storage 'content addressable'
One opportunity which this API opens up is the possibility of making the actual storage of files 'content addressable' . That is, if two users upload the same image for example, only store this file once on disk. This brings benefits in reducing the amount of storage and improving caching (especially in increasingly common situations the moodle data store served from a NFS directory or other remote storage similar). To do this we could use a hash like sha-1 on the file and store the file on disk named by its hash (rather than some arbitary id). Then when someone uploads the same file as has already been uploaded, the hash matches and we just point the database record to the same file on disk. This technique is increasingly being used by enterprise-style repositories as well as things like git. I can see the major benefits in things like scorm packages other such things which have 100's of small duplicate files stored multiple times per package and course, so sometimes you can have the same image file stored 20 different times across 20 different packages in one course, which is then duplicated for multiple classes etcetc. --Dan Poltawski 07:32, 1 June 2008 (CDT)
Excellent idea, Dan, it's in Martin Dougiamas 01:43, 20 June 2008 (CDT)
Specific File Attachments?
We have entries for 'moduleinstance', do we need entries to identify files per other attachments? Such as forum posts, wiki attachments, database attachments, assignment submissions? Mike Churchward 16:27, 9 June 2008 (CDT)
I'm not sure if we need to have such links back to the exact forum post or glossary entry. The idea is that the forum posts (say) would reference the file->id. Would it be useful to have back links too ...? Martin Dougiamas 22:30, 22 June 2008 (CDT)
Squid
Consider proxy support. --Helen Foster 16:53, 9 June 2008 (CDT)
Batch uploads (zips)
Should we require that file API (optionally) require some form of batch file upload or zip/unzip function? Mike Churchward 17:14, 9 June 2008 (CDT)
Yeah, it definitely needs to handle zipped files nicely ... Martin Dougiamas 22:33, 22 June 2008 (CDT)
Skodak's rants
The API should be split into several independent parts:
- File serving API
- file.php
- pluginfile.php
- userfile.php
- rssfile.php
- File storage API
- optional access control
- optional repo sync
- File management API
- File browsing
- File linking (editor integration)
- Upload from repository
File serving API
Deals with serving of files - browser requests file, Moodle sends it back. We have three main files.
file.php
Serves course files. It would be nice to have some special hardcoded protection of backup files - preventing of backup file downloads/uploads; backups contain a lot of personal info, we could block restoring of backups from other sites too.
Implements basic file access. Ideally only images and files linked from course sections should be there, no XSS protection required - we expect javascript, sw, etc. there, no way to make it "secure". The access control is not critical any more if we move most most of the files into modules;
/file.php/courseid/dir/dir/filename.ext
===pluginfile.php=== (aka modfile.php) Sends module, block, question files. Absolute file links need to be rewritten if html editing allowed in module. The links are stored internally as relative links.
- modules decide about access control
- optional XSS protection - student submitted files must not be served with normal headers, we have to force download instead; ideally there should be second wwwroot for serving of untrusted files
/pluginfile.php/contextid/arbitrary/params/or/dirs/filename.ext
pluginfile.php detects the type of plugin from context table, fetches basic info (like $course or $cm if appropriate) and calls plugin function (or later method) which does the access control and finally sends the file to user.
blogs
Blog entries or notes in general do not have context id (because they live in system context). The note attachments are always served with XSS protection on, ideally we should use separate wwwroot for this. Access control can be hardcoded.
/pluginfile.php/SYSCONTEXTID/blog/blogenryid/attachmentname.ext
userfile.php
Personal file storage
- read/write own files only for now
/userfile.php/userid/dir/dir/filename.ext
note: not finished yet ;-)
rssfile.php
Replaces rss/file.php wchi is kept only for backwards compatibility. RSS files should not require sessions/cookies, urls should contain some sort of security token/key Internally the files may be stored in database or together with other files. Etag support should be implemented to improve performance.
/rssfile.php/contextid/any/parameters/module/wants/rss.xml
Again modules and plugins decide what gets sent to user.
