Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

UTF-8 scripts: Difference between revisions

From MoodleDocs
No edit summary
 
(37 intermediate revisions by 5 users not shown)
Line 1: Line 1:
[[UTF-8 migration]] > Recoding PHP scripts
{{obsolete}}[[UTF-8 migration]] > Recoding PHP scripts


==Recoding PHP scripts==
==Recoding PHP scripts==
Line 19: Line 19:


# (D,A) Build one "check for 1.6 upgrade" utility under 1.5. Now it's present in the admin page. It's able to check for BD, PHP and PHP libraries, allowing function execution and built over the new [http://cvs.sourceforge.net/viewcvs.py/moodle/moodle/lib/environmentlib.php?rev=1.11&view=log environmentlib.php] script. All the checks are defined under one simple XML file and one mechanism to update it from http://download.moodle.org has been provided (using the new [http://cvs.sourceforge.net/viewcvs.py/moodle/moodle/lib/componentlib.class.php?rev=1.4&view=log componentlib.class.php] library.
# (D,A) Build one "check for 1.6 upgrade" utility under 1.5. Now it's present in the admin page. It's able to check for BD, PHP and PHP libraries, allowing function execution and built over the new [http://cvs.sourceforge.net/viewcvs.py/moodle/moodle/lib/environmentlib.php?rev=1.11&view=log environmentlib.php] script. All the checks are defined under one simple XML file and one mechanism to update it from http://download.moodle.org has been provided (using the new [http://cvs.sourceforge.net/viewcvs.py/moodle/moodle/lib/componentlib.class.php?rev=1.4&view=log componentlib.class.php] library.
# <span style="color:red">(N, C)</span> datalib.php to support collations under MySQL. This will allow to control language specific ordering from Moodle but 1.6.0 won't offer support for it. Instead, MySQL table/filed collation can be altered "manually" if the default unicode collation isn't enough.
# <span style="color:red">(N, C)</span> POSTPONED: datalib.php to support collations under MySQL. This will allow to control language specific ordering from Moodle but 1.6.0 won't offer support for it. Instead, MySQL table/filed collation can be altered "manually" if the default unicode collation isn't enough.
# (D,A) textlib.class to handle all those utf-compliant functions. It's working since 1.5.3. Based (wrapper) on [http://typo3.org/ Typo3] text handling libraries it offers support for conversion between charsets and a buch of functions like (substr, strtoupper, strpos...).
# (D,A) textlib.class to handle all those utf-compliant functions. It's working since 1.5.3. Based (wrapper) on [http://typo3.org/ Typo3] text handling libraries it offers support for conversion between charsets and a buch of functions like (substr, strtoupper, strpos...).
# <span style="color:red">(N,C)</span> XML import/export (scorm, ims, backup/restore, glossary, quizzes...). Under UTF-8 mode, both utf8_encode() and utf8_decode() won't be needed anymore.
# (D,C) XML import/export (scorm, ims, backup/restore, glossary, quizzes...). Under UTF-8 mode, both utf8_encode() and utf8_decode() won't be needed anymore.
# <span style="color:red">(W,A)</span> Excel Export: A new [http://cvs.sourceforge.net/viewcvs.py/moodle/moodle/lib/excellib.class.php?rev=1.2&view=log excellib.class.php] class wrapper has been built. It works with some [http://pear.php.net/ PEAR] libraries to be able to create UTF-16LE Excel files properly. How this mini-PEAR package will be distributed with Moodle must be decided (directly with the distro, if allowed by PHP people or separate download).
# (D,A) Excel Export: A new [http://cvs.sourceforge.net/viewcvs.py/moodle/moodle/lib/excellib.class.php?rev=1.2&view=log excellib.class.php] class wrapper has been built. It works with some [http://pear.php.net/ PEAR] libraries to be able to create UTF-16LE Excel files properly.
# (D,A) Modify every Excel generator to use the new library. This includes: grade/lib.php, course/grades.php, choice/report.php hotpot/report/default.php, quiz/report/analysis.php, quiz/report/overview.php, quiz/report/responses.ph and survey/download.php
# (D,A) Modify every Excel generator to use the new library. This includes: grade/lib.php, course/grades.php, choice/report.php hotpot/report/default.php, quiz/report/analysis.php, quiz/report/overview.php, quiz/report/responses.ph and survey/download.php
# (D,A) Fixed the break_up_long_words() to work using the textlib functions.
# (D,A) Fixed the break_up_long_words() to work using the textlib functions.
# <span style="color:red">(W,A)</span> Fixed the glossary to find properly the pivot (initial letter) under UTF8 (Skodak has some ideas here about the alphabet for each lang.  
# <span style="color:red">(W,A)</span> Fixed the glossary to find properly the pivot (initial letter) under UTF8 (Skodak has some ideas here about the alphabet for each lang.) Also, for ideogram-based langs, Moodle HQ is planning to do some work (after 1.6). [http://moodle.org/bugs/bug.php?op=show&bugid=6125 Bug 6125]
# (D,A) Modify the rss_title() function to support UTF8 chars.
# (D,A) Modify the rss_title() function to support UTF8 chars.
# <span style="color:red">(W,A)</span> htmlentities() to s() migration everywhere.
# <span style="color:red">(W,A)</span> htmlentities() to s() migration everywhere. [http://moodle.org/bugs/bug.php?op=show&bugid=6121 Bug 6121]
# <span style="color:red">(W,A)</span> uses of substr, strlen, strpos, strtoupper... to use the new textlib class that offers utf8 savvy string manipulation functions.
# <span style="color:red">(W,A)</span> uses of substr, strlen, strpos, strtoupper... to use the new textlib class that offers utf8 savvy string manipulation functions. Some other functions like moodle_strtolower() will be modified to use new text library (done!). They will disappear after 1.6 (do it only in 1.7dev). [http://moodle.org/bugs/bug.php?op=show&bugid=6122 Bug 6122]
# <span style="color:red">(N,A)</span> Modify documentation to let users know how they MUST create their DB before installing Moodle 1.6 explaining all the benefits for being UTF-8 enabled sice the beginning.
# (D,A) Modify documentation to let users know how they MUST create their DB before installing Moodle 1.6 explaining all the benefits for being UTF-8 enabled sice the beginning.
# (D,A) Modify the central installation script to:
# (D,A) Modify the central installation script to:
#* Check DB encoding, warning if unicode hasn't been detected.
#* Check DB encoding, warning if unicode hasn't been detected.
Line 37: Line 37:
#* Execute the environmental checks.
#* Execute the environmental checks.
# (D,A) The wiki module - this is due to the use of htmlentities() without specifying the character set. DFWiki does not have any known problems apparently.
# (D,A) The wiki module - this is due to the use of htmlentities() without specifying the character set. DFWiki does not have any known problems apparently.
# <span style="color:red"> (N,A)</span> UTF-8 national chars can't be used in paths/foldernames. This is not a Moodle problem but simply the limitations of the server. It's highly possible that this problem will persist in Moodle 1.6 as a global solution hasn't been found. With the arrival of DMS, FileManager virtualization and Repositories, it's possible that this issue will become less important...
# (D,A) UTF-8 national chars could not be used in paths/foldernames. Unicode characters in filenames can now be enabled by setting $CFG->unicodecleanfilename=true in config.php, though this option is not recommended. Please note that unicode characters in filenames may be broken during zip/unzip process, native info-zip binaries do not work at all on Windows; please use internal zipping/unzipping.
# <span style="color:red"> (N,A)</span> GD support for UTF-8 strings. Perhaps it'll require some hacking + new fonts to be added (centrally or inside each lang pack).
# (D,A) GD support for UTF-8 strings. Perhaps it'll require some hacking + new fonts to be added (centrally or inside each lang pack). Basically graphlib supported UTF-8 fonts without problems at all. We've updated the central TrueType font and now it supports Latin, Cyrillic and Greek without problems. Some old not-needed fonts have been deleted. And unsupported languages will need to install their language fonts (under moodledata/lang/xx_utf8/fonts).
# (D,C) RSS block. Working fine now with the new texlib.class.php library.
# (D,C) RSS block. Working fine now with the new texlib.class.php library.
# (D,A) Languages list issues, not showing properly lang names. Not action required because now all those names are UTF-8 and they should work properly. I think it's an browser/font issue.
# (D,A) Languages list issues, not showing properly lang names. Not action required because now all those names are UTF-8 and they should work properly. I think it's an browser/font issue.
# (D,C) The Assingment module says that 0 words under some languages without word separators. A new configuration option has been added to count letters instead of words.
# (D,C) The Assingment module says that 0 words under some languages without word separators. A new configuration option has been added to count letters instead of words.
# <span style="color:red"> (W,C)</span> Modify the email_to_user() function to enable encoding of mails based on site setting and/or user preference. This is a must because a lot of mail clients/widgets don't support UTF-8 encodings.
# (D,C) Modify the email_to_user() function to enable encoding of mails based on site setting and/or user preference. This is a must because a lot of mail clients/widgets don't support UTF-8 encodings. Done (email encoding now can be specified at site and user level plus support any charset (header + body).
# <span style="color:red">(N,C)</span> Look for all the occurrences of "en" or 'en' is source code and change them to 'en_utf8' is necessary.
# <span style="color:red">(N,C)</span> Look for all the occurrences of "en" or 'en' is source code and change them to 'en_utf8' is necessary. [http://moodle.org/bugs/bug.php?op=show&bugid=6124 Bug 6124]
# <span style="color:red">(W,A)</span> New lang edition interface for utf_8 lang packs. It should support dataroot lang packs, use 'en_utf8' as master language and "lock" the new langconfig.php file.
# (D,A) New lang edition interface for utf_8 lang packs. It should support dataroot lang packs, use 'en_utf8' as master language and "lock" the new langconfig.php file.
# <span style="color:red">(W,C)</span> DB Migration with this functionalities:
# (D,C) DB Migration with this functionalities:
#* Convert all the users/courses/site languages to their new alternative.
#* Convert all the users/courses/site languages to their new alternative.
#* Able to be executed from command line.
#* Recover from crash smoothly.
# <span style="color:red">(N,C)</span> Modify the document_file() function to work properly (links to docs.moodle.org)
#* Handle and central tables and official modules. Contrib modules should implement their own script. See [[UTF-8 contrib]] for more info.
# <span style="color:red">(N,C)</span> Modify the footer output to allow it to go to the proper (en, es...) wiki page in https://docs.moodle.org. A list of available languages can be harcoded, defaulting to en.
# <strike><span style="color:red">(N,C)</span> Modify the document_file() function to work properly (links to docs.moodle.org). If documentation isn't going to be included with Moodle anymore, such function will disappear completely in a near future.</strike>
# <span style="color:red">(N,C)</span> Review the RSS feeds creator to detect if conversion to UTF-8 is needed.
# (D,A) Modify the footer output to allow it to go to the proper (en, es...) wiki page in https://docs.moodle.org. A list of available languages can be harcoded, defaulting to en.
# <span style="color:red">(N,C)</span> Analyse if we need the $SESSION->encoding. Delete if not necessary.
# (D,C) Review the RSS feeds creator to detect if conversion to UTF-8 is needed.
# <span style="color:red">(W,A)</span> Delete all the get/print_string('thisencoding') and change them to the new current_charset() function.
# (D,C) Analyse if we need the $SESSION->encoding. Removed from main CVS, 3rd party plugins must be updated anyway to use curent_curset() and $CFG->unicodedb to be fully utf8 and 1.6 compatible.
# <span style="color:red">(N,C)</span> Analyse and, if possible, implement a bit improved moodle_set_locale() function because of differences between *nix locales and Win32 locales. They should go to a new string inside each langconfig.php file and, after OS detection use the correct one.
# (D,A) Delete all the get/print_string('thisencoding') and change them to the new current_charset() function.Seems finished.
# <span style="color:red">(W,C)</span> Implement one PEAR download utility if finally we cannot add it to standard distro (it's ready since some time ago under http://download.moodle.org/pear). Needed to generate Excel files and, potentially, it'll grow. Add it to credits.
# (D,C) Analyse and, if possible, implement a bit improved moodle_setlocale() function because of [[Table of locales|differences between Unix locales and Win32 locales]]. They should go to a new string inside each langconfig.php file and, after OS detection use the correct one. Done, 95% of Windows langs will work.
# <span style="color:red">(N,C)</span> Handling of passwords. After DB conversion, passwords containing non ASCII chars will be broken. So we should be able to check them again, but converted to the original encoding. If the second check returns true then we can safely the stored password hash for that user/course..... (every lang includes one 'oldcharset' string to know to what encoding we must convert the entered password).
# (D,C) Implement one PEAR download utility if finally we cannot add it to standard distro (it's ready since some time ago under http://download.moodle.org/pear). Needed to generate Excel files and, potentially, it'll grow. Finally we got perms from Xavier Noguer (see lib/pear/README.txt for a note about the PHP license and Moodle) and it has been included. Anyway, the pear download continues being generated in download.moodle.org daily.
# <span style="color:red">(N,A)</span> Modify (and potentially upgrade) the MyPHPAdmin module in order to recognise new lang pack names.
# (D,C) Handling of passwords. After DB conversion, password hashes are updated to utf8 during next user login. First is tried hash of unicode plain text password, then the plain text is converted into 'oldcharset' defined in the new language pack and its hash is checked again.
# <span style="color:red">(N,C)</span>Add one new parameter to authentication methods in order to be able to convert from external sources (ldap, db...) encoding to utf8 if Moodle is running in that mode avoiding the current utf8_decode() implemented.
# (D,A) Modify (and potentially upgrade) the MyPHPAdmin module in order to recognise new lang pack names. Done: just support to utf8 langs added.
# <span style="color:red">(N,C)</span> POSTPONED: Add one new parameter (xxx_original_encoding) to authentication methods in order to be able to convert from external sources (ldap, db...) encoding to utf8 if Moodle is running in that mode avoiding the current utf8_decode() implemented. [http://moodle.org/bugs/bug.php?op=show&bugid=6123 Bug 6123]
# (D,A) Modify the install.php script to be able to detect DB encoding and warn about it plus use the new environment stuff to perform tests.
# (D,A) Create a collection of '''installer.php''' (to avoid conflicts with old install.php) files to be stored under the install/lang directory and to be used exclusively in the installation process. Hack get_string() to support this files ONLY in installation and make a script to be able to build them daily from contents existing in other language files (i.e, no manual handling of them!). Also, perhaps, add the possibility of language download at the end of the installation script.


== Patches reference ==
== Patches reference ==
Line 78: Line 81:
Many, or even most end-users (including myself = Tim) are not sufficiently confident making extensive patches, so our Moodles have been garbling in important areas (email/excel).
Many, or even most end-users (including myself = Tim) are not sufficiently confident making extensive patches, so our Moodles have been garbling in important areas (email/excel).


One interesting PHP-UTF8 reference: http://www.phpwact.org/php/i18n/utf-8#mail_functions
Great reference for Unicode Fonts: http://www.unifont.org/fontguide/ and http://www.travelphrases.info/fonts.html
For item #8: may be for letter-index use "distinct" first letters from existing data, and not lang alphabet? it can help in multilanguage glossary... --[[User:Ne Nashev|Ne Nashev]] 13:53, 29 March 2006 (WST) 


[[Category:UTF-8|Scripts]]
[[Category:UTF-8|Scripts]]

Latest revision as of 14:53, 10 November 2013

Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

UTF-8 migration > Recoding PHP scripts

Recoding PHP scripts

Introduction

This page will show a list of well known UTF-8 related modifications to be applied to 1.6 in order to work better under Unicode. Some of the changes can be applied always while others would break compatibility with non UTF-8 sites so they have to be executed conditionally, i.e:

     if (!empty($CFG->unicodedb)) {
         //Code to be executed in UTF8 mode
     } else {
         //Old code
     }

For each modification we'll show if it's executed always or conditionally, using the "A" and "C" abbreviation. Also, the current status will be maintained with "N"ot implemented, "W"ork in progress and "D"one. Obviously, "D" is the desired final status for all the modifications.

Finally, please, you are welcome to add new items to the list, thanks!, but do it at the end (to maintain its current numeration).

The List

  1. (D,A) Build one "check for 1.6 upgrade" utility under 1.5. Now it's present in the admin page. It's able to check for BD, PHP and PHP libraries, allowing function execution and built over the new environmentlib.php script. All the checks are defined under one simple XML file and one mechanism to update it from http://download.moodle.org has been provided (using the new componentlib.class.php library.
  2. (N, C) POSTPONED: datalib.php to support collations under MySQL. This will allow to control language specific ordering from Moodle but 1.6.0 won't offer support for it. Instead, MySQL table/filed collation can be altered "manually" if the default unicode collation isn't enough.
  3. (D,A) textlib.class to handle all those utf-compliant functions. It's working since 1.5.3. Based (wrapper) on Typo3 text handling libraries it offers support for conversion between charsets and a buch of functions like (substr, strtoupper, strpos...).
  4. (D,C) XML import/export (scorm, ims, backup/restore, glossary, quizzes...). Under UTF-8 mode, both utf8_encode() and utf8_decode() won't be needed anymore.
  5. (D,A) Excel Export: A new excellib.class.php class wrapper has been built. It works with some PEAR libraries to be able to create UTF-16LE Excel files properly.
  6. (D,A) Modify every Excel generator to use the new library. This includes: grade/lib.php, course/grades.php, choice/report.php hotpot/report/default.php, quiz/report/analysis.php, quiz/report/overview.php, quiz/report/responses.ph and survey/download.php
  7. (D,A) Fixed the break_up_long_words() to work using the textlib functions.
  8. (W,A) Fixed the glossary to find properly the pivot (initial letter) under UTF8 (Skodak has some ideas here about the alphabet for each lang.) Also, for ideogram-based langs, Moodle HQ is planning to do some work (after 1.6). Bug 6125
  9. (D,A) Modify the rss_title() function to support UTF8 chars.
  10. (W,A) htmlentities() to s() migration everywhere. Bug 6121
  11. (W,A) uses of substr, strlen, strpos, strtoupper... to use the new textlib class that offers utf8 savvy string manipulation functions. Some other functions like moodle_strtolower() will be modified to use new text library (done!). They will disappear after 1.6 (do it only in 1.7dev). Bug 6122
  12. (D,A) Modify documentation to let users know how they MUST create their DB before installing Moodle 1.6 explaining all the benefits for being UTF-8 enabled sice the beginning.
  13. (D,A) Modify the central installation script to:
    • Check DB encoding, warning if unicode hasn't been detected.
    • Execute the environmental checks.
  14. (D,A) Modify the Windows32 Complete Package installation script to:
    • Force DB creation under UTF-8.
    • Execute the environmental checks.
  15. (D,A) The wiki module - this is due to the use of htmlentities() without specifying the character set. DFWiki does not have any known problems apparently.
  16. (D,A) UTF-8 national chars could not be used in paths/foldernames. Unicode characters in filenames can now be enabled by setting $CFG->unicodecleanfilename=true in config.php, though this option is not recommended. Please note that unicode characters in filenames may be broken during zip/unzip process, native info-zip binaries do not work at all on Windows; please use internal zipping/unzipping.
  17. (D,A) GD support for UTF-8 strings. Perhaps it'll require some hacking + new fonts to be added (centrally or inside each lang pack). Basically graphlib supported UTF-8 fonts without problems at all. We've updated the central TrueType font and now it supports Latin, Cyrillic and Greek without problems. Some old not-needed fonts have been deleted. And unsupported languages will need to install their language fonts (under moodledata/lang/xx_utf8/fonts).
  18. (D,C) RSS block. Working fine now with the new texlib.class.php library.
  19. (D,A) Languages list issues, not showing properly lang names. Not action required because now all those names are UTF-8 and they should work properly. I think it's an browser/font issue.
  20. (D,C) The Assingment module says that 0 words under some languages without word separators. A new configuration option has been added to count letters instead of words.
  21. (D,C) Modify the email_to_user() function to enable encoding of mails based on site setting and/or user preference. This is a must because a lot of mail clients/widgets don't support UTF-8 encodings. Done (email encoding now can be specified at site and user level plus support any charset (header + body).
  22. (N,C) Look for all the occurrences of "en" or 'en' is source code and change them to 'en_utf8' is necessary. Bug 6124
  23. (D,A) New lang edition interface for utf_8 lang packs. It should support dataroot lang packs, use 'en_utf8' as master language and "lock" the new langconfig.php file.
  24. (D,C) DB Migration with this functionalities:
    • Convert all the users/courses/site languages to their new alternative.
    • Recover from crash smoothly.
    • Handle and central tables and official modules. Contrib modules should implement their own script. See UTF-8 contrib for more info.
  25. (N,C) Modify the document_file() function to work properly (links to docs.moodle.org). If documentation isn't going to be included with Moodle anymore, such function will disappear completely in a near future.
  26. (D,A) Modify the footer output to allow it to go to the proper (en, es...) wiki page in https://docs.moodle.org. A list of available languages can be harcoded, defaulting to en.
  27. (D,C) Review the RSS feeds creator to detect if conversion to UTF-8 is needed.
  28. (D,C) Analyse if we need the $SESSION->encoding. Removed from main CVS, 3rd party plugins must be updated anyway to use curent_curset() and $CFG->unicodedb to be fully utf8 and 1.6 compatible.
  29. (D,A) Delete all the get/print_string('thisencoding') and change them to the new current_charset() function.Seems finished.
  30. (D,C) Analyse and, if possible, implement a bit improved moodle_setlocale() function because of differences between Unix locales and Win32 locales. They should go to a new string inside each langconfig.php file and, after OS detection use the correct one. Done, 95% of Windows langs will work.
  31. (D,C) Implement one PEAR download utility if finally we cannot add it to standard distro (it's ready since some time ago under http://download.moodle.org/pear). Needed to generate Excel files and, potentially, it'll grow. Finally we got perms from Xavier Noguer (see lib/pear/README.txt for a note about the PHP license and Moodle) and it has been included. Anyway, the pear download continues being generated in download.moodle.org daily.
  32. (D,C) Handling of passwords. After DB conversion, password hashes are updated to utf8 during next user login. First is tried hash of unicode plain text password, then the plain text is converted into 'oldcharset' defined in the new language pack and its hash is checked again.
  33. (D,A) Modify (and potentially upgrade) the MyPHPAdmin module in order to recognise new lang pack names. Done: just support to utf8 langs added.
  34. (N,C) POSTPONED: Add one new parameter (xxx_original_encoding) to authentication methods in order to be able to convert from external sources (ldap, db...) encoding to utf8 if Moodle is running in that mode avoiding the current utf8_decode() implemented. Bug 6123
  35. (D,A) Modify the install.php script to be able to detect DB encoding and warn about it plus use the new environment stuff to perform tests.
  36. (D,A) Create a collection of installer.php (to avoid conflicts with old install.php) files to be stored under the install/lang directory and to be used exclusively in the installation process. Hack get_string() to support this files ONLY in installation and make a script to be able to build them daily from contents existing in other language files (i.e, no manual handling of them!). Also, perhaps, add the possibility of language download at the end of the installation script.

Patches reference

Post here all the solutions you know if you consider that they'll be interesting to solve some of the pending items in the list above.


Patches to allow Japanese Language Moodles that function without garbling have been prepared and are available at at the following sites.

These patches are explained here http://moodle.org/mod/forum/discuss.php?d=13558

Mr. Kashiwagi's "Supertak" Patch (described in the thread above) http://www.supertak.com/down/sample.htm

Prof Kita's patches and rpms (based in part on Mr. Kashiwagi's) http://t-kita.net/rpm/FC/moodle/
A read me file in English describing the rpm http://t-kita.net/rpm/moodle/README-rpm-en.txt A patch describing all the things that need to be done http://t-kita.net/rpm/FC/moodle/patches/moodle-t-kita.patch Many, or even most end-users (including myself = Tim) are not sufficiently confident making extensive patches, so our Moodles have been garbling in important areas (email/excel).

One interesting PHP-UTF8 reference: http://www.phpwact.org/php/i18n/utf-8#mail_functions

Great reference for Unicode Fonts: http://www.unifont.org/fontguide/ and http://www.travelphrases.info/fonts.html

For item #8: may be for letter-index use "distinct" first letters from existing data, and not lang alphabet? it can help in multilanguage glossary... --Ne Nashev 13:53, 29 March 2006 (WST)