Perth Hackfest October 2012/File Storage/Cloud
- What do you mean about cloud computing?
- Bunch of vms that you turn on or off depending on your load
- Related services
- Make file storage options pluggable
- Amazon EBS
- Talk about different storage strategies
- Problems with big clusters
- Session locking
- File API
- Right now, forced to use local storage. Cannot take advantage of cloud storage options
- To make File API more pluggable
- Do audit to look at all instances in which $CFG->dataroot is used directly
- Make File API a pluggable interface
- Write API to allow creation cloud storage integration
- Future
- Want to make dataroot completely on cloud storage. Seems dangerous and will break a lot of things.
- Problems?
- Do cloud storage options do proper access control?
- Can generate token for one user or a group of users. Token can be one use or expires
- Do cloud storage options do proper access control?
- Links:
- Sam Marshall’s blog post about “Moodle filesystem size and repositories”: http://learn.open.ac.uk/mod/oublog/viewpost.php?post=123877
- Memcache session tracker issue http://tracker.moodle.org/browse/MDL-31501
- Session locking
- When you have data in a session that needs to be there, then you have problems if sessions aren’t locked
- e.g. one session writes data and another session blows that away
- Opening up 20 moodle tabs can also lock up apache processes (mini DoS attack)
- Sessions can be up to 6 MB (should be small)
- Solutions
- With MUC, then sessions can be lightweight
- lots of stuff stored in cache isn’t being used in every request
- some stuff cannot be in MUC, like admin tree
- Closing session earlier on a long request
- example: downloading a css file, should close session before serving file
- Petr suggested investigate no session locking for guest sessions
- Have memcache or redis to store sessions
- Moodlerooms already written plugins for memcache and redis
- With MUC, then sessions can be lightweight
- When you have data in a session that needs to be there, then you have problems if sessions aren’t locked
- Contributors
- Matt
- Ash
- Petr
- Kris
Audit of $CFG->dataroot usage in Moodle core:
(Note: this audit assumes that any usage of dataroot that doesn’t need to be shared amongst nodes is fine. ie, it is assumed a writeable data root will be available on all hosts for caching/temp purposes)
admin/cli/install.php: numerous uses of $CFG->dataroot, but looks fine.
admin/environment.php: find out what this is? component installer? might need some work
admin/handlevirus.php: antivirus stuff might not work. needs testing. not used by any clients.
admin/settings/location.php, iplookup/tests/geoip_test.php: GeoIP information needs to move elsewhere (or go on each node).
admin/tool/dbtransfer/: probably doesn't matter. we don't use it. refers to climaintenance.html but we don't use that either & don't need to use it (use LB for outage).
admin/tool/langimport/index.php: language importing. needs to be fixed to have shared lang state. OR we disabled it and commit all lang packs.
admin/tool/unittest/: unit tests will run on a single node, so no problems.
auth/shibboleth/logout.php: stupid disk sessions stuff. theres a note there that it needs to be rewritten. ignore.
backup/, batchbackup, batchrestore/: temp backup stuff. fine.
enrol/flatfile/lib.php, enrol/imsenterprise/lib.php: default ims enrol xml path is in $CFG->dataroot but can be configured to be elsewhere.
filter/algebra/, filter/tex/: should just work. looks like a cache. should move to cache dir. needs testing.
lib/adminlib.php: no problem
lib/dml/moodle_database.php: cache. fine.
lib/dmllib.php: stupid, but all good.
lib/environmentlib.php: once off upon install, fine.
lib/filelib.php: investigate. do we need persistent curl cookies? (curl_cookies.txt)
lib/graphlib.php: needs patching if lang directory changes.
lib/pdflib.php: not used.
lib/phpunit/: unit tests will run on one node - fine.
lib/sessionlib.php: disk sessions not used.
lib/setup.php: fine, note that moodledata climaintenance.html will not work (we don't use that though, use the CFG setting instead or LB outage)
lib/setuplib.php: fine
lib/simpletest/broken_testfiltermanager.php: this is a unit test. ignore. plus, it looks ok.
lib/tests/: more unit tests. need to review lang test if we change lang destination.
lib/uploadlib.php: tested and works fine.
mod/data/lib.php: either lock it down to mod/data/presets/, or store shared templates elsewhere such as db, memcached, or a filearea.
mod/lightboxgallery/mod_form.php:135: why is this here?
question/engine/upgrade/logger.php: fine
repository/filesystem/lib.php: this is for the "filesystem" repository which gets mounted in moodledata/repository/ - fine.
user/grouppix.php: groups have images??? investigate. if so, needs to be fixed to use filearea.