Development talk:File API: Difference between revisions

Latest revision as of 06:26, 2 February 2009

Main tasks

File Storage API:
- abstract (M1)
- local pool implementation (M1)
- DB schema (M1)
- deletion, acls, metadata (M1)
- problem: empty directories, file overwriting
File Manager API:
- unique class, able to handle one "file area" (M2)
- security (M2)
- hack old file manager to be able to work with new fileareas (M3)
- js and non js implementations of FileManager (M4)
- integration with editor (M4)
- integration with formslib (M4)
- integration with repos (M4)
- problem: zip support
File Serving:
- from pool:
  - file.php
  - pluginfile.php
  - draftfile.php
  - userfile.php
- from other moddata places:
  - rssfile.php
  - user/pix.php
  - user/pixgroup.php
Migration:
- course files (as much as possible, allow fallback) (M2)
- moddata
Backup & restore:

Milestones

M1: File storage API completed (this week) M2: migration of course files + new filephp + FileManager + hacked old file manager (next monday) M3: ...

rssfile.php

It's said that rss/file.php is kept only for backwards compatibility. But what exactly is meant by "backwards compatibility"? Just display feeds with nice error messages and info how to re-subscribe?

meta information

It seems like it would be nice to include some meta information about certain file types, e.g.:

images: width, height, alt
flash files: width, height

Regarding: Development:File_API#Files_database_tables

Perhaps we could also hash filepath and filename and index by them, to save some text limitations in the DB side (length limits of indexes, not indexable, complex retrieval...). Eloy Lafuente (stronk7) 11:54, 29 June 2008 (CDT)

Also, perhaps we should store finally the plugin type there to save some queries per request, using it to drive to the correct file handling of each plugin. Eloy Lafuente (stronk7) 18:54, 29 June 2008 (CDT)

@@ Line 1: / Line 1: @@
-Some quick questions to avoid forgetting them:
+===Main tasks===
+* File Storage API:
+** abstract (M1)
+** local pool implementation (M1)
+** DB schema (M1)
+** deletion, acls, metadata (M1)
+** problem: empty directories, file overwriting
+* File Manager API:
+** unique class, able to handle one "file area" (M2)
+** security (M2)
+** hack old file manager to be able to work with new fileareas (M3)
+** js and non js implementations of FileManager (M4)
+** integration with editor (M4)
+** integration with formslib (M4)
+** integration with repos (M4)
+** problem: zip support
+* File Serving:
+** from pool:
+*** file.php
+*** pluginfile.php
+*** draftfile.php
+*** userfile.php
+** from other moddata places:
+*** rssfile.php
+*** user/pix.php
+*** user/pixgroup.php
+* Migration:
+** course files (as much as possible, allow fallback) (M2)
+** moddata
+**
+* Backup & restore:
-) Will them be under the control of FileAPI (or, as they are now, fixed local storage)?
+===Milestones===
+M1: File storage API completed (this week)
+M2: migration of course files + new filephp + FileManager + hacked old file manager (next monday)
+M3: ...
-- dataroot/temp
-- dataroot/lang
-- dataroot/cache
-- dataroot/environment
-- dataroot/filter
-- dataroot/rss
-- dataroot/search
-- dataroot/sessions
-- dataroot/upgradelogs
-[[User:Martin Dougiamas|Martin Dougiamas]] 03:20, 28 April 2008 (CDT) : I don't see these as being in the API - I've updated the spec.
-) Assuming we'll have a cool OOP FileAPI...
+===rssfile.php===
+It's said that rss/file.php is kept only for backwards compatibility. But what exactly is meant by "backwards compatibility"?
+Just display feeds with nice error messages and info how to re-subscribe?
-- a) Will it support different FileAPI classes (to be able to store in other systems) ?
+== meta information ==
-- b) Will it support multiple FileAPI classes working together (like the Repo) ?
-[[User:Martin Dougiamas|Martin Dougiamas]] 03:20, 28 April 2008 (CDT) : Hmm, I suppose it makes sense to switch the backend from local file storage to specify something else (eg database storage) but multiple File storage places doesn't make sense to me, that is the Repository API and the Portfolio API.
+It seems like it would be nice to include some meta information about certain file types, e.g.:
-) I've annotated in red some things that have sounded strange in my first look.
+* images: width, height, alt
+* flash files: width, height
-[[User:Martin Dougiamas|Martin Dougiamas]] 03:20, 28 April 2008 (CDT) : Thanks, all fixed.
+Regarding: [[Development:File_API#Files_database_tables]]
-) Are we going to have "directory records" in the implementation, or that is going to be handled exclusively by the "moodlepath" column?
+::Perhaps we could also hash filepath and filename and index by them, to save some text limitations in the DB side (length limits of indexes, not indexable, complex retrieval...). [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 11:54, 29 June 2008 (CDT)
-[[User:Martin Dougiamas|Martin Dougiamas]] 03:20, 28 April 2008 (CDT) : Good point.  I was thinking of moodlepath only but I wonder if directory records might be more efficient.  New table, I guess.
+::Also, perhaps we should store finally the plugin type there to save some queries per request, using it to drive to the correct file handling of each plugin. [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 18:54, 29 June 2008 (CDT)
-) One general question... are we going to "force" all modules to be "autocontained" ? How are we going to handle resources, for example (with all those css, links, images..). In general how are we going to handle multiple-file packages?
-[[User:Martin Dougiamas|Martin Dougiamas]] 03:20, 28 April 2008 (CDT) : they'll be a set of files, probably in a "directory" specified by moodlepath of directory record.  What problems do you see?  Should we retain better knowledge of the original group of files?
-That's all for now, ciao4niao :-) [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 21:27, 5 April 2008 (CDT)
-== Making Storage 'content addressable' ==
-One opportunity  which this API opens up is the possibility of making the actual storage of files 'content addressable' . That is, if two users upload the same image for example, only store this file once on disk. This  brings benefits in reducing the amount of storage and improving caching (especially in increasingly common situations the moodle data store served from a NFS directory or other remote storage similar). To do this we could use a hash like sha-1 on the file and store the file on disk named by its hash (rather than some arbitary id). Then when someone uploads the same file as has already been uploaded, the hash matches and we just point the database record to the same file on disk. This technique is increasingly being used by enterprise-style repositories as well as things like git. I can see the major benefits in things like scorm packages other such things which have 100's of small duplicate files stored multiple times per package and course, so sometimes you can have the same image file stored 20 different times across 20 different packages in one course, which is then duplicated for multiple classes etcetc. --[[User:Dan Poltawski|Dan Poltawski]] 07:32, 1 June 2008 (CDT)
-== Specific File Attachments? ==
-We have entries for 'moduleinstance', do we need entries to identify files per other attachments? Such as forum posts, wiki attachments, database attachments, assignment submissions?

Documentation