UTF-8 scripts: Difference between revisions

Revision as of 10:26, 24 March 2006

UTF-8 migration > Recoding PHP scripts

Recoding PHP scripts

Introduction

This page will show a list of well known UTF-8 related modifications to be applied to 1.6 in order to work better under Unicode. Some of the changes can be applied always while others would break compatibility with non UTF-8 sites so they have to be executed conditionally, i.e:

     if (!empty($CFG->unicodedb)) {
         //Code to be executed in UTF8 mode
     } else {
         //Old code
     }

For each modification we'll show if it's executed always or conditionally, using the "A" and "C" abbreviation. Also, the current status will be maintained with "N"ot implemented, "W"ork in progress and "D"one. Obviously, "D" is the desired final status for all the modifications.

Finally, please, you are welcome to add new items to the list, thanks!, but do it at the end (to maintain its current numeration).

The List

(D,A) Build one "check for 1.6 upgrade" utility under 1.5. Now it's present in the admin page. It's able to check for BD, PHP and PHP libraries, allowing function execution and built over the new environmentlib.php script. All the checks are defined under one simple XML file and one mechanism to update it from http://download.moodle.org has been provided (using the new componentlib.class.php library.
(N, C) datalib.php to support collations under MySQL. This will allow to control language specific ordering from Moodle but 1.6.0 won't offer support for it. Instead, MySQL table/filed collation can be altered "manually" if the default unicode collation isn't enough.
(D,A) textlib.class to handle all those utf-compliant functions. It's working since 1.5.3. Based (wrapper) on Typo3 text handling libraries it offers support for conversion between charsets and a buch of functions like (substr, strtoupper, strpos...).
(N,C) XML import/export (scorm, ims, backup/restore, glossary, quizzes...). Under UTF-8 mode, both utf8_encode() and utf8_decode() won't be needed anymore.
(D,A) Excel Export: A new excellib.class.php class wrapper has been built. It works with some PEAR libraries to be able to create UTF-16LE Excel files properly.
(D,A) Modify every Excel generator to use the new library. This includes: grade/lib.php, course/grades.php, choice/report.php hotpot/report/default.php, quiz/report/analysis.php, quiz/report/overview.php, quiz/report/responses.ph and survey/download.php
(D,A) Fixed the break_up_long_words() to work using the textlib functions.
(W,A) Fixed the glossary to find properly the pivot (initial letter) under UTF8 (Skodak has some ideas here about the alphabet for each lang.
(D,A) Modify the rss_title() function to support UTF8 chars.
(W,A) htmlentities() to s() migration everywhere.
(W,A) uses of substr, strlen, strpos, strtoupper... to use the new textlib class that offers utf8 savvy string manipulation functions.
(N,A) Modify documentation to let users know how they MUST create their DB before installing Moodle 1.6 explaining all the benefits for being UTF-8 enabled sice the beginning.
(D,A) Modify the central installation script to:
- Check DB encoding, warning if unicode hasn't been detected.
- Execute the environmental checks.
(D,A) Modify the Windows32 Complete Package installation script to:
- Force DB creation under UTF-8.
- Execute the environmental checks.
(D,A) The wiki module - this is due to the use of htmlentities() without specifying the character set. DFWiki does not have any known problems apparently.
(D,A) UTF-8 national chars could not be used in paths/foldernames. Unicode characters in filenames can now be enabled by setting $CFG->unicodecleanfilename=true in config.php, though this option is not recommended. Please note that unicode characters in filenames may be broken during zip/unzip process, native info-zip binaries do not work at all on Windows; please use internal zipping/unzipping.
(N,A) GD support for UTF-8 strings. Perhaps it'll require some hacking + new fonts to be added (centrally or inside each lang pack).
(D,C) RSS block. Working fine now with the new texlib.class.php library.
(D,A) Languages list issues, not showing properly lang names. Not action required because now all those names are UTF-8 and they should work properly. I think it's an browser/font issue.
(D,C) The Assingment module says that 0 words under some languages without word separators. A new configuration option has been added to count letters instead of words.
(D,C) Modify the email_to_user() function to enable encoding of mails based on site setting and/or user preference. This is a must because a lot of mail clients/widgets don't support UTF-8 encodings. Done (email encoding now can be specified at site and user level plus support any charset (header + body).
(N,C) Look for all the occurrences of "en" or 'en' is source code and change them to 'en_utf8' is necessary.
(W,A) New lang edition interface for utf_8 lang packs. It should support dataroot lang packs, use 'en_utf8' as master language and "lock" the new langconfig.php file.
(W,C) DB Migration with this functionalities:
- Convert all the users/courses/site languages to their new alternative.
- Able to be executed from command line.
(N,C) Modify the document_file() function to work properly (links to docs.moodle.org)
(N,C) Modify the footer output to allow it to go to the proper (en, es...) wiki page in https://docs.moodle.org. A list of available languages can be harcoded, defaulting to en.
(N,C) Review the RSS feeds creator to detect if conversion to UTF-8 is needed.
(D,C) Analyse if we need the $SESSION->encoding. Removed from main CVS, 3rd party plugins must be updated anyway to use curent_curset() and $CFG->unicodedb to be fully utf8 and 1.6 compatible.
(D,A) Delete all the get/print_string('thisencoding') and change them to the new current_charset() function.Seems finished.
(N,C) Analyse and, if possible, implement a bit improved moodle_setlocale() function because of Table of locales. They should go to a new string inside each langconfig.php file and, after OS detection use the correct one.
(W,C) Implement one PEAR download utility if finally we cannot add it to standard distro (it's ready since some time ago under http://download.moodle.org/pear). Needed to generate Excel files and, potentially, it'll grow. Add it to credits.
(D,C) Handling of passwords. After DB conversion, password hashes are updated to utf8 during next user login. First is tried hash of unicode plain text password, then the plain text is converted into 'oldcharset' defined in the new language pack and its hash is checked again.
(D,A) Modify (and potentially upgrade) the MyPHPAdmin module in order to recognise new lang pack names. Done: just support to utf8 langs added.
(N,C) Add one new parameter to authentication methods in order to be able to convert from external sources (ldap, db...) encoding to utf8 if Moodle is running in that mode avoiding the current utf8_decode() implemented.
(D,A) Modify the install.php script to be able to detect DB encoding and warn about it plus use the new environment stuff to perform tests.
(D,A) Create a collection of installer.php (to avoid conflicts with old install.php) files to be stored under the install/lang directory and to be used exclusively in the installation process. Hack get_string() to support this files ONLY in installation and make a script to be able to build them daily from contents existing in other language files (i.e, no manual handling of them!). Also, perhaps, add the possibility of language download at the end of the installation script.

Patches reference

Post here all the solutions you know if you consider that they'll be interesting to solve some of the pending items in the list above.

Patches to allow Japanese Language Moodles that function without garbling have been prepared and are available at at the following sites.

These patches are explained here http://moodle.org/mod/forum/discuss.php?d=13558

Mr. Kashiwagi's "Supertak" Patch (described in the thread above) http://www.supertak.com/down/sample.htm

Prof Kita's patches and rpms (based in part on Mr. Kashiwagi's) http://t-kita.net/rpm/FC/moodle/
A read me file in English describing the rpm http://t-kita.net/rpm/moodle/README-rpm-en.txt A patch describing all the things that need to be done http://t-kita.net/rpm/FC/moodle/patches/moodle-t-kita.patch Many, or even most end-users (including myself = Tim) are not sufficiently confident making extensive patches, so our Moodles have been garbling in important areas (email/excel).

One interesting PHP-UTF8 reference: http://www.phpwact.org/php/i18n/utf-8#mail_functions

@@ Line 53: / Line 53: @@
 # (D,C) Analyse if we need the $SESSION->encoding. Removed from main CVS, 3rd party plugins must be updated anyway to use curent_curset() and $CFG->unicodedb to be fully utf8 and 1.6 compatible.
 # (D,A) Delete all the get/print_string('thisencoding') and change them to the new current_charset() function.Seems finished.
-# <span style="color:red">(N,C)</span> Analyse and, if possible, implement a bit improved moodle_setlocale() function because of [[differences between *nix locales and Win32 locales|Table of locales]]. They should go to a new string inside each langconfig.php file and, after OS detection use the correct one.
+# <span style="color:red">(N,C)</span> Analyse and, if possible, implement a bit improved moodle_setlocale() function because of [[differences between Unix locales and Win32 locales|Table of locales]]. They should go to a new string inside each langconfig.php file and, after OS detection use the correct one.
 # <span style="color:red">(W,C)</span> Implement one PEAR download utility if finally we cannot add it to standard distro (it's ready since some time ago under http://download.moodle.org/pear). Needed to generate Excel files and, potentially, it'll grow. Add it to credits.
 # (D,C) Handling of passwords. After DB conversion, password hashes are updated to utf8 during next user login. First is tried hash of unicode plain text password, then the plain text is converted into 'oldcharset' defined in the new language pack and its hash is checked again.

Documentation