Development:File API - Revision history

TimHunt: /* See also */

2008-11-06T06:56:52Z

Tsala: /* See also */ tracker issue link

2008-09-22T08:27:21Z

Skodak: /* File pool details */

2008-09-17T13:57:29Z

File pool details

← Older revision		Revision as of 13:57, 17 September 2008
Line 457:		Line 457:
	...		...
	/'''2d'''		/'''2d'''
	/231e2dc421be4fcd0172e5afceea3970e2f3d940~~.jpg~~		/231e2dc421be4fcd0172e5afceea3970e2f3d940
	...		...
	/fe		/fe

Skodak: /* Integration with htmleditor */

2008-08-29T07:29:30Z

Integration with htmleditor

← Older revision		Revision as of 07:29, 29 August 2008
Line 481:		Line 481:
	There are several problems here:		There are several problems here:
	# when adding new resource its context does not exist yet, we will have to create some table to handle temporary file storage for adding of new stuff, not easy but should be solvable - maybe we could abuse the course context id or store it temporarily in some special user file area		# when adding new resource its context does not exist yet, we will have to create some table to handle temporary file storage for adding of new stuff, not easy but should be solvable - maybe we could abuse the course context id or store it temporarily in some special user file area
	# we can not use absolute address relinking for pluginfile.php links, instead we can use the absolute links only when editing and before storage convert them to something like @@thispluginfile~~/intro~~@@/2112/112/image.jpg before storage. the local links would be converted to full absolute links before display or editing. Not all file areas will support this (ex: linking to assignment submission does not make sense because nobody else may access it anyway). This would allow us to implement image preview in html editor.		# we can not use absolute address relinking for pluginfile.php links, instead we can use the absolute links only when editing and before storage convert them to something like @@thispluginfile@@/2112/112/image.jpg before storage. the local links would be converted to full absolute links before display or editing. Not all file areas will support this (ex: linking to assignment submission does not make sense because nobody else may access it anyway). This would allow us to implement image preview in html editor.

	Html editor should contain simplified single pane file manager with basic operations only - select file area, browse file area, upload file/copy user file/use repo file, delete. The editor will communicate with modules and core through ajax call to some script specified by module embedding the editor. The callback script would use different logic to construct the tree of files than the File manager, it needs to know only about files that other ppl viewing the resulting html may access.		Html editor should contain simplified single pane file manager with basic operations only - select file area, browse file area, upload file/copy user file/use repo file, delete. The editor will communicate with modules and core through ajax call to some script specified by module embedding the editor. The callback script would use different logic to construct the tree of files than the File manager, it needs to know only about files that other ppl viewing the resulting html may access.

Skodak: /* zip support */

2008-08-06T08:56:55Z

zip support

← Older revision		Revision as of 08:56, 6 August 2008
Line 546:		Line 546:

	===zip support===		===zip support===
	Zip format is an old standard for compression of files, it was created long before the unicode and ~~it was never standardised to~~ support it. There are several ways used for encoding of non-ascii characters in path names, unfortunately it is not standardised ~~at all~~. Most windows packers use DOS encoding.		Zip format is an old standard for compression of files, it was created long before the unicode and added support for it just recently. There are several ways used for encoding of non-ascii characters in path names, unfortunately it is not much standardised. Most windows packers use DOS encoding.

	Client software:		Client software:

Skodak: /* unicode zip support */

2008-08-06T08:34:39Z

unicode zip support

← Older revision		Revision as of 08:34, 6 August 2008
Line 545:		Line 545:
	List of hard to solve prolbems		List of hard to solve prolbems

	===~~unicode~~ zip support===		===zip support===
	~~Unicode chars in zip files uploaded by teachers - unfortunately there~~ is ~~no 100% solution that will work~~ for ~~anybody because most zip programs do not~~ support ~~unicode~~, it is ~~usually ''garbage in/garbage out'' which works in some cases only~~		Zip format is an old standard for compression of files, it was created long before the unicode and it was never standardised to support it. There are several ways used for encoding of non-ascii characters in path names, unfortunately it is not standardised at all. Most windows packers use DOS encoding.

	~~Latest WinZIP 11.2 and Total Commander seem to support some very limited form of utf~~-~~8 encodings of file names. I managed to create a zip file~~ in Windows (~~Czech locale~~) ~~and extract them in linux with native PHP zip functions~~, ~~the only step I needed to add was conversion cp852~~(DOS ~~charset for Czech locale~~) -~~> UTF~~-8. ~~The native windows zipping did not work for me though~~, ~~because it does~~ some ~~different borking of charsets.~~		Client software:
			* Windows built-in compression - bundled with Windows, non-standard DOS encoding only
			* WinZip - shareware, unicode option (since v11.2)
			* TotalCommander - shareware, single byte(DOS) encoding only
			* 7-Zip - free, unicode or DOS encoding depending on characters used in file name (since v4.58beta)
			* Info-ZIP - free, uses some weird charset conversions

	~~In any case it seems likely that native~~ PHP support in PHP 5.2.x ~~should~~ be ~~better than current pclzip~~ or ~~infozip binary.~~		PHP extraction:
			* Info-ZIP binary execution - no unicode support at all, mangles charsets in filename (depends on OS, see docs), files must be copied to temp directory before compression and after extraction
			* PclZip PHP library - reads single byte encoded names only, problems with random problems and higher memory usage
			* Zip PHP extension - reads single byte encoded names only, 64bit operating system can not open/create archives with more than 500 files (depends on sum of lengths of all filenames and directories, to be fixed in PHP 5.3 and external PECL library, no PHP 5.2.x backport planned!), adding of files is limited by number of free file handles (around 1000 - depends on OS and other PHP code, workaround is to close and reopen archive)

			Large file support:
			PHP running under 32bit operating systems does not support files >2GB (do not expect fix before PHP 6). This might be a potential problem for larger backups.

			Tar Alternative:
			* tar with gzip compression - easy to implement in PHP + zlib extension (PclTar, Tar from PEAR or custom code)
			* no problem with unicode in *nix, Windows again expects DOS encoding :-(
			* seems suitable for backup/restore - yay!

			Roadmap:
			# add zip processing class that fully hides the underlying library
			# use single byte encoding "garbage in/garbage out" approach for encoding of files in zip archives; add new 'zipencoding' string into lang packs (ex: cp852 DOS charset for Czech locale) and use it during extraction, we might support true unicode later when PHP Zip extension does that

	===empty directories===		===empty directories===

Stronk7 at 17:04, 25 July 2008

2008-07-25T17:04:22Z

← Older revision		Revision as of 17:04, 25 July 2008
Line 1:		Line 1:
	This page outlines the current thinking about implementing file storage and access in Moodle 2.0. It's a SPECIFICATION UNDER CONSTRUCTION!		{{Moodle_2.0}}This page outlines the current thinking about implementing file storage and access in Moodle 2.0. It's a SPECIFICATION UNDER CONSTRUCTION!

	The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document. However, if you have questions, problems to report or major changes to suggest please add them to the [[Development_talk:File_API\|page comments]], or start a discussion in the [http://moodle.org/mod/forum/view.php?id=1807 Repositories forum]. We'll endeavour to merge all such suggestions into the main spec before we start development.		The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document. However, if you have questions, problems to report or major changes to suggest please add them to the [[Development_talk:File_API\|page comments]], or start a discussion in the [http://moodle.org/mod/forum/view.php?id=1807 Repositories forum]. We'll endeavour to merge all such suggestions into the main spec before we start development.

Skodak: /* file overwriting */

2008-07-20T23:02:54Z

file overwriting

← Older revision		Revision as of 23:02, 20 July 2008
Line 559:		Line 559:
	===file overwriting===		===file overwriting===
	Concept of file overwriting does not exist anymore here, the path+filename are not enforced to be unique - we can not make index because sloppy mssql does not allow indexes larger than 900 bytes :-( We will haev to emulate it somehow and deal with collisions if found.		Concept of file overwriting does not exist anymore here, the path+filename are not enforced to be unique - we can not make index because sloppy mssql does not allow indexes larger than 900 bytes :-( We will haev to emulate it somehow and deal with collisions if found.

			Note: solved with unique index on pathnamehash field.

	== Some little comments to be considered (to avoid forgetting them) ==		== Some little comments to be considered (to avoid forgetting them) ==

Skodak: /* empty directories */

2008-07-20T23:02:13Z

empty directories

← Older revision		Revision as of 23:02, 20 July 2008
Line 554:		Line 554:
	===empty directories===		===empty directories===
	Hmm, thinking a bit more about Justin's comment I realised there is no support for empty directories in this proposal. This will require either new table or some hack in files table - maybe we could add files with "." as name and just skip them when iterating directory content.		Hmm, thinking a bit more about Justin's comment I realised there is no support for empty directories in this proposal. This will require either new table or some hack in files table - maybe we could add files with "." as name and just skip them when iterating directory content.

			Note: solved with files named '.'; the dirs are automatically created when adding files, the area root dir is created when browsing.

	===file overwriting===		===file overwriting===

Skodak: /* files_cleanup table */

2008-07-20T22:59:35Z

files_cleanup table

← Older revision		Revision as of 22:59, 20 July 2008
Line 283:		Line 283:

	\|-		\|-
	\|~~sha1hash~~		\|contenthash
	\|varchar(40)		\|varchar(40)
	\|		\|

← Older revision		Revision as of 06:56, 6 November 2008
Line 636:		Line 636:
	==See also==		==See also==

			* [[Development:Using the file API]]
	* [[Development:Repository API]]		* [[Development:Repository API]]
	* [[Development:Portfolio API]]		* [[Development:Portfolio API]]
	* MDL-14589 - File API Meta issue		* MDL-14589 - File API Meta issue

← Older revision		Revision as of 08:27, 22 September 2008
Line 638:		Line 638:
	* [[Development:Repository API]]		* [[Development:Repository API]]
	* [[Development:Portfolio API]]		* [[Development:Portfolio API]]
			* MDL-14589 - File API Meta issue