File API internals - Revision history

Devdocsbot: Protected "File API internals": Developer Docs Migration ([Edit=Allow only administrators] (indefinite))

2022-05-06T07:27:03Z

Protected "File API internals": Developer Docs Migration ([Edit=Allow only administrators] (indefinite))

← Older revision	Revision as of 07:27, 6 May 2022
(No difference)

Devdocsbot: Update migration status and path

2022-04-28T07:01:07Z

Update migration status and path

← Older revision		Revision as of 07:01, 28 April 2022
Line 1:		Line 1:
	{{Template:Migrated\|newDocId=/docs/apis/files/internals}}		{{Template:Migrated\|newDocId=/docs/apis/subsystems/files/internals}}
	{{Infobox Project		{{Infobox Project
	\|name = File API		\|name = File API

Sarjona: Update migration status and path

2022-04-20T14:11:36Z

Update migration status and path

← Older revision		Revision as of 14:11, 20 April 2022
Line 1:		Line 1:
			{{Template:Migrated\|newDocId=/docs/apis/files/internals}}
	{{Infobox Project		{{Infobox Project
	\|name = File API		\|name = File API

Mudrd8mz: Text replacement - "class="nicetable"" to "class="wikitable""

2021-07-14T13:22:32Z

Text replacement - "class="nicetable"" to "class="wikitable""

← Older revision		Revision as of 13:22, 14 July 2021
Line 78:		Line 78:
	If, for example, the same image is used in a user's profile, and a forum post, then there will be two rows in this table, one for each use of the file, and Moodle will treat the two as separate files, even though the file is only stored once on disc.		If, for example, the same image is used in a user's profile, and a forum post, then there will be two rows in this table, one for each use of the file, and Moodle will treat the two as separate files, even though the file is only stored once on disc.

	{\| class="~~nicetable~~"		{\| class="wikitable"
	! Field		! Field
	! Type		! Type

Mudrd8mz: Text replacement - "" to ""

2021-07-14T13:09:56Z

Text replacement - "</code>" to "</syntaxhighlight>"

← Older revision		Revision as of 13:09, 14 July 2021
Line 190:		Line 190:
	'''Each plugin may directly access only files in own context and areas!'''		'''Each plugin may directly access only files in own context and areas!'''

	Low level access API is defined in ''file_storage'' class which is obtained from <syntaxhighlight lang="php">get_file_storage()</~~code~~>.		Low level access API is defined in ''file_storage'' class which is obtained from <syntaxhighlight lang="php">get_file_storage()</syntaxhighlight>.

	====Storing a file====		====Storing a file====
Line 219:		Line 219:
	/file.php/courseid/dir/dir/filename.ext		/file.php/courseid/dir/dir/filename.ext

	Internally the files are stored in <syntaxhighlight lang="php">array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</~~code~~>		Internally the files are stored in <syntaxhighlight lang="php">array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</syntaxhighlight>

	The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more.		The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more.
Line 266:		Line 266:
	/pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext		/pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext

	Internally stored in <syntaxhighlight lang="php">array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</~~code~~>		Internally stored in <syntaxhighlight lang="php">array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</syntaxhighlight>

	/pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext		/pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext

	Internally stored in <syntaxhighlight lang="php">array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</~~code~~>		Internally stored in <syntaxhighlight lang="php">array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</syntaxhighlight>

Mudrd8mz: Text replacement - "`" to ""`


2021-07-14T13:02:53Z
Text replacement - "<code>" to "<syntaxhighlight lang="php">"

				
				
				
				
				
				← Older revision
				Revision as of 13:02, 14 July 2021
				
Line 190:
Line 190:
'''Each plugin may directly access only files in own context and areas!''' '''Each plugin may directly access only files in own context and areas!'''



Low level access API is defined in ''file_storage'' class which is obtained from <code>get_file_storage()</code>. Low level access API is defined in ''file_storage'' class which is obtained from <syntaxhighlight lang="php">get_file_storage()</code>.



====Storing a file==== ====Storing a file====
Line 219:
Line 219:
  /file.php/courseid/dir/dir/filename.ext   /file.php/courseid/dir/dir/filename.ext



Internally the files are stored in <code>array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</code> Internally the files are stored in <syntaxhighlight lang="php">array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</code>



The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more. The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more.
Line 266:
Line 266:
  /pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext   /pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext



Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</code> Internally stored in <syntaxhighlight lang="php">array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</code>



  /pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext   /pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext



Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</code> Internally stored in <syntaxhighlight lang="php">array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</code>



Demsley: /* File storage on disk */
2017-12-19T13:28:11Z
File storage on disk

				
				
				
				
				
				← Older revision
				Revision as of 13:28, 19 December 2017
				
Line 51:
Line 51:
Files are stored according to the SHA1 hash of their content. This means each file with particular contents is stored once, irrespective of how many times it is included in different places, even if it is referred to by different names. (This idea comes from the git version control system.) To relate a file on disc to a user-comprehensible path or filename, you need to use the ''files'' database table. See the next section. Files are stored according to the SHA1 hash of their content. This means each file with particular contents is stored once, irrespective of how many times it is included in different places, even if it is referred to by different names. (This idea comes from the git version control system.) To relate a file on disc to a user-comprehensible path or filename, you need to use the ''files'' database table. See the next section.



Suppose a file has SHA1 hash 081371cb102fa559e81993fddc230c79205232ce. Then it will be stored in on disc as moodledata/filedir/08/13/081371cb102fa559e81993fddc230c79205232ce. Suppose a file has SHA1 hash 081371cb102fa559e81993fddc230c79205232ce in the files table contenthash field. Then it will be stored in on disc as moodledata/filedir/08/13/081371cb102fa559e81993fddc230c79205232ce.



This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash. This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.



James.mclean: /* File storage on disk */
2014-11-25T02:53:56Z
File storage on disk

				
				
				
				
				
				← Older revision
				Revision as of 02:53, 25 November 2014
				
Line 57:
Line 57:
As files on-disk are named with their sha1 hash, there is a simple way of validating files have not been corrupted since upload by using the 'sha1sum' command available in most GNU/Linux distributions.   As files on-disk are named with their sha1 hash, there is a simple way of validating files have not been corrupted since upload by using the 'sha1sum' command available in most GNU/Linux distributions.  
In the below example, the user changed directory to the known location of a file on disk (sourced by finding the contenthash of the relevant file in the mdl_files database table).   In the below example, the user changed directory to the known location of a file on disk (sourced by finding the contenthash of the relevant file in the mdl_files database table).  
Then, in this directory the 'sha1sum' command is issued with the file that you wish to hash the content of. Returned from the command is the hash of the content (on the left) and the file name (on the right). Where a file is NOT corrupted after upload, these two strings will match. Where a file IS corrupted after upload, these will differ. Then, in this directory the 'sha1sum' command is issued with the file that you wish to hash the content of. Returned from the command is the hash of the content (on the left) and the file name (on the right).  
 
Where a file is NOT corrupted after upload, these two strings will match.  



   $ cd /moodlepath/moodledata/filedir/1d/df/    $ cd /moodlepath/moodledata/filedir/1d/df/
Line 63:
Line 65:
   1ddf5b375fcb74929cdd7efda4f47efc61414edf  1ddf5b375fcb74929cdd7efda4f47efc61414edf    1ddf5b375fcb74929cdd7efda4f47efc61414edf  1ddf5b375fcb74929cdd7efda4f47efc61414edf



Where a file IS corrupted after upload, these will differ:

  $ cd /moodlepath/moodledata/filedir/42/32/
  $ sha1sum 42327aac8ce5741f51f42be298fa63686fe81b7a
  9442188152c02f65267103d78167d122c87002cd  42327aac8ce5741f51f42be298fa63686fe81b7a
This is a very handy trick as in the case of any disk corruption (shared storage issues, hard drive issues, disk sector issues etc) the corrupted files can be detected without resorting to manually comparing to previous backups. This is a very handy trick as in the case of any disk corruption (shared storage issues, hard drive issues, disk sector issues etc) the corrupted files can be detected without resorting to manually comparing to previous backups.






James.mclean: /* File storage on disk */
2014-11-25T02:48:43Z
File storage on disk

				
				
				
				
				
				← Older revision
				Revision as of 02:48, 25 November 2014
				
Line 54:
Line 54:



This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash. This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.

As files on-disk are named with their sha1 hash, there is a simple way of validating files have not been corrupted since upload by using the 'sha1sum' command available in most GNU/Linux distributions. 
In the below example, the user changed directory to the known location of a file on disk (sourced by finding the contenthash of the relevant file in the mdl_files database table). 
Then, in this directory the 'sha1sum' command is issued with the file that you wish to hash the content of. Returned from the command is the hash of the content (on the left) and the file name (on the right). Where a file is NOT corrupted after upload, these two strings will match. Where a file IS corrupted after upload, these will differ.

  $ cd /moodlepath/moodledata/filedir/1d/df/
  $ sha1sum 1ddf5b375fcb74929cdd7efda4f47efc61414edf
  1ddf5b375fcb74929cdd7efda4f47efc61414edf  1ddf5b375fcb74929cdd7efda4f47efc61414edf

This is a very handy trick as in the case of any disk corruption (shared storage issues, hard drive issues, disk sector issues etc) the corrupted files can be detected without resorting to manually comparing to previous backups.



=== Files table === === Files table ===



Quen: /* Tar packer */
2013-11-27T13:53:48Z
Tar packer

				
				
				
				
				
				← Older revision
				Revision as of 13:53, 27 November 2013
				
Line 402:
Line 402:
=== Tar packer === === Tar packer ===



A .tar.gz format packer is available from Moodle 2.6 (requires zlib extension) and can be selected for use in Moodle backup via an experimental option. A .tar.gz format packer is available from Moodle 2.6 (requires zlib extension) and can be selected for use in Moodle backup via an experimental option. MDL-41838.



The packer is currently limited to ASCII filenames and individual files are limited to 8GB each, but unlike zip there is no limit on the total filesize. It uses the old POSIX format and is compatible with GNU tar using default options. The packer is currently limited to ASCII filenames and individual files are limited to 8GB each, but unlike zip there is no limit on the total filesize. It uses the old POSIX format and is compatible with GNU tar using default options.

← Older revision		Revision as of 13:02, 14 July 2021
Line 190:		Line 190:
	'''Each plugin may directly access only files in own context and areas!'''		'''Each plugin may directly access only files in own context and areas!'''

	Low level access API is defined in ''file_storage'' class which is obtained from <~~code~~>get_file_storage()</code>.		Low level access API is defined in ''file_storage'' class which is obtained from <syntaxhighlight lang="php">get_file_storage()</code>.

	====Storing a file====		====Storing a file====
Line 219:		Line 219:
	/file.php/courseid/dir/dir/filename.ext		/file.php/courseid/dir/dir/filename.ext

	Internally the files are stored in <~~code~~>array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</code>		Internally the files are stored in <syntaxhighlight lang="php">array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</code>

	The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more.		The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more.
Line 266:		Line 266:
	/pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext		/pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext

	Internally stored in <~~code~~>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</code>		Internally stored in <syntaxhighlight lang="php">array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</code>

	/pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext		/pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext

	Internally stored in <~~code~~>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</code>		Internally stored in <syntaxhighlight lang="php">array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</code>

← Older revision		Revision as of 13:28, 19 December 2017
Line 51:		Line 51:
	Files are stored according to the SHA1 hash of their content. This means each file with particular contents is stored once, irrespective of how many times it is included in different places, even if it is referred to by different names. (This idea comes from the git version control system.) To relate a file on disc to a user-comprehensible path or filename, you need to use the ''files'' database table. See the next section.		Files are stored according to the SHA1 hash of their content. This means each file with particular contents is stored once, irrespective of how many times it is included in different places, even if it is referred to by different names. (This idea comes from the git version control system.) To relate a file on disc to a user-comprehensible path or filename, you need to use the ''files'' database table. See the next section.

	Suppose a file has SHA1 hash 081371cb102fa559e81993fddc230c79205232ce. Then it will be stored in on disc as moodledata/filedir/08/13/081371cb102fa559e81993fddc230c79205232ce.		Suppose a file has SHA1 hash 081371cb102fa559e81993fddc230c79205232ce in the files table contenthash field. Then it will be stored in on disc as moodledata/filedir/08/13/081371cb102fa559e81993fddc230c79205232ce.

	This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.		This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.

← Older revision		Revision as of 02:53, 25 November 2014
Line 57:		Line 57:
	As files on-disk are named with their sha1 hash, there is a simple way of validating files have not been corrupted since upload by using the 'sha1sum' command available in most GNU/Linux distributions.		As files on-disk are named with their sha1 hash, there is a simple way of validating files have not been corrupted since upload by using the 'sha1sum' command available in most GNU/Linux distributions.
	In the below example, the user changed directory to the known location of a file on disk (sourced by finding the contenthash of the relevant file in the mdl_files database table).		In the below example, the user changed directory to the known location of a file on disk (sourced by finding the contenthash of the relevant file in the mdl_files database table).
	Then, in this directory the 'sha1sum' command is issued with the file that you wish to hash the content of. Returned from the command is the hash of the content (on the left) and the file name (on the right). Where a file is NOT corrupted after upload, these two strings will match~~. Where a file IS corrupted after upload, these will differ~~.		Then, in this directory the 'sha1sum' command is issued with the file that you wish to hash the content of. Returned from the command is the hash of the content (on the left) and the file name (on the right).

			Where a file is NOT corrupted after upload, these two strings will match.

	$ cd /moodlepath/moodledata/filedir/1d/df/		$ cd /moodlepath/moodledata/filedir/1d/df/
Line 63:		Line 65:
	1ddf5b375fcb74929cdd7efda4f47efc61414edf 1ddf5b375fcb74929cdd7efda4f47efc61414edf		1ddf5b375fcb74929cdd7efda4f47efc61414edf 1ddf5b375fcb74929cdd7efda4f47efc61414edf

			Where a file IS corrupted after upload, these will differ:

			$ cd /moodlepath/moodledata/filedir/42/32/
			$ sha1sum 42327aac8ce5741f51f42be298fa63686fe81b7a
			9442188152c02f65267103d78167d122c87002cd 42327aac8ce5741f51f42be298fa63686fe81b7a
	This is a very handy trick as in the case of any disk corruption (shared storage issues, hard drive issues, disk sector issues etc) the corrupted files can be detected without resorting to manually comparing to previous backups.		This is a very handy trick as in the case of any disk corruption (shared storage issues, hard drive issues, disk sector issues etc) the corrupted files can be detected without resorting to manually comparing to previous backups.

← Older revision		Revision as of 02:48, 25 November 2014
Line 54:		Line 54:

	This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.		This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.

			As files on-disk are named with their sha1 hash, there is a simple way of validating files have not been corrupted since upload by using the 'sha1sum' command available in most GNU/Linux distributions.
			In the below example, the user changed directory to the known location of a file on disk (sourced by finding the contenthash of the relevant file in the mdl_files database table).
			Then, in this directory the 'sha1sum' command is issued with the file that you wish to hash the content of. Returned from the command is the hash of the content (on the left) and the file name (on the right). Where a file is NOT corrupted after upload, these two strings will match. Where a file IS corrupted after upload, these will differ.

			$ cd /moodlepath/moodledata/filedir/1d/df/
			$ sha1sum 1ddf5b375fcb74929cdd7efda4f47efc61414edf
			1ddf5b375fcb74929cdd7efda4f47efc61414edf 1ddf5b375fcb74929cdd7efda4f47efc61414edf

			This is a very handy trick as in the case of any disk corruption (shared storage issues, hard drive issues, disk sector issues etc) the corrupted files can be detected without resorting to manually comparing to previous backups.

	=== Files table ===		=== Files table ===

← Older revision		Revision as of 13:53, 27 November 2013
Line 402:		Line 402:
	=== Tar packer ===		=== Tar packer ===

	A .tar.gz format packer is available from Moodle 2.6 (requires zlib extension) and can be selected for use in Moodle backup via an experimental option.		A .tar.gz format packer is available from Moodle 2.6 (requires zlib extension) and can be selected for use in Moodle backup via an experimental option. MDL-41838.

	The packer is currently limited to ASCII filenames and individual files are limited to 8GB each, but unlike zip there is no limit on the total filesize. It uses the old POSIX format and is compatible with GNU tar using default options.		The packer is currently limited to ASCII filenames and individual files are limited to 8GB each, but unlike zip there is no limit on the total filesize. It uses the old POSIX format and is compatible with GNU tar using default options.