Perth Hackfest October 2012/File Storage/Cloud

• What do you mean about cloud computing?
  • Bunch of vms that you turn on or off depending on your load
  • Related services
• Make file storage options pluggable
  • Amazon EBS
• Talk about different storage strategies
• Problems with big clusters
  • Session locking
  • File API
    • Right now, forced to use local storage. Cannot take advantage of cloud storage options
• To make File API more pluggable
  • Do audit to look at all instances in which $CFG->dataroot is used directly
  • Make File API a pluggable interface
  • Write API to allow creation cloud storage integration
  • Future
    • Want to make dataroot completely on cloud storage. Seems dangerous and will break a lot of things.
• Problems?
  • Do cloud storage options do proper access control?
    • Can generate token for one user or a group of users. Token can be one use or expires
• Links:
  • Sam Marshall’s blog post about “Moodle filesystem size and repositories”: http://learn.open.ac.uk/mod/oublog/viewpost.php?post=123877
  • Memcache session tracker issue http://tracker.moodle.org/browse/MDL-31501
• Session locking
  • When you have data in a session that needs to be there, then you have problems if sessions aren’t locked
    • e.g. one session writes data and another session blows that away
  • Opening up 20 moodle tabs can also lock up apache processes (mini DoS attack)
  • Sessions can be up to 6 MB (should be small)
  • Solutions
    • With MUC, then sessions can be lightweight
      • lots of stuff stored in cache isn’t being used in every request
      • some stuff cannot be in MUC, like admin tree
    • Closing session earlier on a long request
      • example: downloading a css file, should close session before serving file
    • Petr suggested investigate no session locking for guest sessions
    • Have memcache or redis to store sessions
      • Moodlerooms already written plugins for memcache and redis
• Contributors
  • Matt
  • Ash
  • Petr
  • Kris

Audit of $CFG->dataroot usage in Moodle core:

(Note: this audit assumes that any usage of dataroot that doesn’t need to be shared amongst nodes is fine. ie, it is assumed a writeable data root will be available on all hosts for caching/temp purposes)

admin/cli/install.php: numerous uses of $CFG->dataroot, but looks fine.

admin/environment.php: find out what this is? component installer? might need some work

admin/handlevirus.php: antivirus stuff might not work. needs testing. not used by any clients.

admin/settings/location.php, iplookup/tests/geoip_test.php: GeoIP information needs to move elsewhere (or go on each node).

admin/tool/dbtransfer/: probably doesn't matter. we don't use it. refers to climaintenance.html but we don't use that either & don't need to use it (use LB for outage).

admin/tool/langimport/index.php: language importing. needs to be fixed to have shared lang state. OR we disabled it and commit all lang packs.

admin/tool/unittest/: unit tests will run on a single node, so no problems.

auth/shibboleth/logout.php: stupid disk sessions stuff. theres a note there that it needs to be rewritten. ignore.

backup/, batchbackup, batchrestore/: temp backup stuff. fine.

enrol/flatfile/lib.php, enrol/imsenterprise/lib.php: default ims enrol xml path is in $CFG->dataroot but can be configured to be elsewhere.

filter/algebra/, filter/tex/: should just work. looks like a cache. should move to cache dir. needs testing.

lib/adminlib.php: no problem

lib/dml/moodle_database.php: cache. fine.

lib/dmllib.php: stupid, but all good.

lib/environmentlib.php: once off upon install, fine.

lib/filelib.php: investigate. do we need persistent curl cookies? (curl_cookies.txt)

lib/graphlib.php: needs patching if lang directory changes.

lib/pdflib.php: not used.

lib/phpunit/: unit tests will run on one node - fine.

lib/sessionlib.php: disk sessions not used.

lib/setup.php: fine, note that moodledata climaintenance.html will not work (we don't use that though, use the CFG setting instead or LB outage)

lib/setuplib.php: fine

lib/simpletest/broken_testfiltermanager.php: this is a unit test. ignore. plus, it looks ok.

lib/tests/: more unit tests. need to review lang test if we change lang destination.

lib/uploadlib.php: tested and works fine.

mod/data/lib.php: either lock it down to mod/data/presets/, or store shared templates elsewhere such as db, memcached, or a filearea.

mod/lightboxgallery/mod_form.php:135: why is this here?

question/engine/upgrade/logger.php: fine

repository/filesystem/lib.php: this is for the "filesystem" repository which gets mounted in moodledata/repository/ - fine.

user/grouppix.php: groups have images??? investigate. if so, needs to be fixed to use filearea.