Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

UTF-8 scripts

From MoodleDocs

UTF-8 migration > Recoding PHP scripts

This page is under construction!!

Only some preliminary ideas have been defined.

Recoding PHP scripts

Introduction

This page will show a list of well known UTF-8 related modifications to be applied to 1.6 in order to work better under Unicode. Some of the changes can be applied always while others would break compatibility with non UTF-8 sites so they have to be executed conditionally, i.e:

     if (!empty($CFG->unicodedb)) {
         //Code to be executed in UTF8 mode
     } else {
         //Old code
     }

For each modification we'll show if it's executed always or conditionally, using the "A" and "C" abbreviation. Also, the current status will be maintained with "N"ot implemented, "W"ork in progress and "D"one. Obviously, "D" is the desired final status for all the modifications.

Finally, please, you are welcome to add new items to the list, thanks!, but do it at the end (to maintain its current numeration).

The List

  1. (D,A) Build one "check for 1.6 upgrade" utility under 1.5. Now it's present in the admin page. It's able to check for BD, PHP and PHP libraries, allowing function execution and built over the new environmentlib.php script. All the checks are defined under one simple XML file and one mechanism to update it from http://download.moodle.org has been provided (using the new componentlib.class.php library.
  2. (N, C) datalib.php to support collations under MySQL. This will allow to control language specific ordering from Moodle but 1.6.0 won't offer support for it. Instead, MySQL table/filed collation can be altered "manually" if the default unicode collation isn't enough.
  3. (D,A) textlib.class to handle all those utf-compliant functions. It's working since 1.5.3. Based (wrapper) on Typo3 text handling libraries it offers support for conversion between charsets and a buch of functions like (substr, strtoupper, strpos...).
  4. (N,C) XML import/export (scorm, ims, backup/restore, glossary, quizzes...). Under UTF-8 mode, both utf8_encode() and utf8_decode() won't be needed anymore.
  5. (W,A) Excel Export: A new excellib.class.php class wrapper has been built. It works with some PEAR libraries to be able to create UTF-16LE Excel files properly. How this mini-PEAR package will be distributed with Moodle must be decided (directly with the distro, if allowed by PHP people or separate download).
  6. (W,A) htmlentities() to s() migration everywhere.
  7. (W,A)uses of substr, strlen, strpos, strtoupper... to use the new textlib class that offers utf8 savvy string manipulation functions.
  8. (N,A) Modify documentation to let users know how they MUST create their DB before installing Moodle 1.6 explaining all the benefits for being UTF-8 enabled sice the beginning.
  9. (D,A) Modify the central installation script to:
    • Check DB encoding, warning if unicode hasn't been detected.
    • Execute the environmental checks.
  10. (D,A) Modify the Windows32 Complete Package installation script to:
    • Force DB creation under UTF-8.
    • Execute the environmental checks.
  11. (D,A) The wiki module - this is due to the use of htmlentities() without specifying the character set. DFWiki does not have any known problems apparently.
  12. (N,A)UTF-8 national chars can't be used in paths/foldernames. This is not a Moodle problem but simply the limitations of the server. It's highly possible that this problem will persist in Moodle 1.6 as a global solution hasn't been found. With the arrival of DMS, FileManager virtualization and Repositories, it's possible that this issue will become less important...
  13. (N,A)GD support for UTF-8 strings. Perhaps it'll require some hacking + new fonts to be added (centrally or inside each lang pack).
  14. (D,A) RSS block. Working fine now with the new texlib.class.php library.


6) The language list drop down menu does not display properly. I am not sure if it should, and I don't really care (since I normally limit my site to a few languages) but it looks like this http://ds21.cc.yamaguchi-u.ac.jp/~econo/temp/languagemenu.jpg

7) The Assingment module says that 0 Words have been submitted after a Japanese language submission, because there are no spaces in Japanese.

De-UTF8-ing for client side applications

One of the biggest problems which remains after the move to UTF-8 is that while UTF-8 is great on the web a lot of client side software in Japan and China is *NOT* UTF-8 compatible.

The Japanese community's solution to this (provided originally by Mr. Kashiwagi, below) is: 1) At the inferface between Moodle and client side software (particularly email clients and spread sheet programs) Moodle checks to see if there is a /lib folder in the current language. 2) If a lang/xyx/lib folder is present, then the routines in that folder are used to convert the encoding to formats compatible with client side software. ( E.g. Even outlook expresses is not compatible with UTF8 in the subject line, and Excel can not deal with UTF8 either, so the lang/lib/ files contains code to covert the UTF8 to a client readable format. ) Contact with client side software occurs at the following points. 2.1) Email sent to Email clients 2.2) Grade Files export to Excel 2.3) Quizes and lessons imported from text editors

3) If the lang/xyz/lib folder is not present, then UTF8 encoding is used to talk to the client software as normal.

Patches reference

Patches to allow Japanese Language Moodles that function without garbling have been prepared and are available at at the following sites.

These patches are explained here http://moodle.org/mod/forum/discuss.php?d=13558

Mr. Kashiwagi's "Supertak" Patch (described in the thread above) http://www.supertak.com/down/sample.htm

Prof Kita's patches and rpms (based in part on Mr. Kashiwagi's) http://t-kita.net/rpm/FC/moodle/
A read me file in English describing the rpm http://t-kita.net/rpm/moodle/README-rpm-en.txt A patch describing all the things that need to be done http://t-kita.net/rpm/FC/moodle/patches/moodle-t-kita.patch Many, or even most end-users (including myself = Tim) are not sufficiently confident making extensive patches, so our Moodles have been garbling in important areas (email/excel).