Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

UTF-8 scripts

From MoodleDocs
Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

UTF-8 migration > Recoding PHP scripts

Recoding PHP scripts

Introduction

This page will show a list of well known UTF-8 related modifications to be applied to 1.6 in order to work better under Unicode. Some of the changes can be applied always while others would break compatibility with non UTF-8 sites so they have to be executed conditionally, i.e:

     if (!empty($CFG->unicodedb)) {
         //Code to be executed in UTF8 mode
     } else {
         //Old code
     }

For each modification we'll show if it's executed always or conditionally, using the "A" and "C" abbreviation. Also, the current status will be maintained with "N"ot implemented, "W"ork in progress and "D"one. Obviously, "D" is the desired final status for all the modifications.

Finally, please, you are welcome to add new items to the list, thanks!, but do it at the end (to maintain its current numeration).

The List

  1. (D,A) Build one "check for 1.6 upgrade" utility under 1.5. Now it's present in the admin page. It's able to check for BD, PHP and PHP libraries, allowing function execution and built over the new environmentlib.php script. All the checks are defined under one simple XML file and one mechanism to update it from http://download.moodle.org has been provided (using the new componentlib.class.php library.
  2. (N, C) POSTPONED: datalib.php to support collations under MySQL. This will allow to control language specific ordering from Moodle but 1.6.0 won't offer support for it. Instead, MySQL table/filed collation can be altered "manually" if the default unicode collation isn't enough.
  3. (D,A) textlib.class to handle all those utf-compliant functions. It's working since 1.5.3. Based (wrapper) on Typo3 text handling libraries it offers support for conversion between charsets and a buch of functions like (substr, strtoupper, strpos...).
  4. (D,C) XML import/export (scorm, ims, backup/restore, glossary, quizzes...). Under UTF-8 mode, both utf8_encode() and utf8_decode() won't be needed anymore.
  5. (D,A) Excel Export: A new excellib.class.php class wrapper has been built. It works with some PEAR libraries to be able to create UTF-16LE Excel files properly.
  6. (D,A) Modify every Excel generator to use the new library. This includes: grade/lib.php, course/grades.php, choice/report.php hotpot/report/default.php, quiz/report/analysis.php, quiz/report/overview.php, quiz/report/responses.ph and survey/download.php
  7. (D,A) Fixed the break_up_long_words() to work using the textlib functions.
  8. (W,A) Fixed the glossary to find properly the pivot (initial letter) under UTF8 (Skodak has some ideas here about the alphabet for each lang.) Also, for ideogram-based langs, Moodle HQ is planning to do some work (after 1.6). Bug 6125
  9. (D,A) Modify the rss_title() function to support UTF8 chars.
  10. (W,A) htmlentities() to s() migration everywhere. Bug 6121
  11. (W,A) uses of substr, strlen, strpos, strtoupper... to use the new textlib class that offers utf8 savvy string manipulation functions. Some other functions like moodle_strtolower() will be modified to use new text library (done!). They will disappear after 1.6 (do it only in 1.7dev). Bug 6122
  12. (D,A) Modify documentation to let users know how they MUST create their DB before installing Moodle 1.6 explaining all the benefits for being UTF-8 enabled sice the beginning.
  13. (D,A) Modify the central installation script to:
    • Check DB encoding, warning if unicode hasn't been detected.
    • Execute the environmental checks.
  14. (D,A) Modify the Windows32 Complete Package installation script to:
    • Force DB creation under UTF-8.
    • Execute the environmental checks.
  15. (D,A) The wiki module - this is due to the use of htmlentities() without specifying the character set. DFWiki does not have any known problems apparently.
  16. (D,A) UTF-8 national chars could not be used in paths/foldernames. Unicode characters in filenames can now be enabled by setting $CFG->unicodecleanfilename=true in config.php, though this option is not recommended. Please note that unicode characters in filenames may be broken during zip/unzip process, native info-zip binaries do not work at all on Windows; please use internal zipping/unzipping.
  17. (D,A) GD support for UTF-8 strings. Perhaps it'll require some hacking + new fonts to be added (centrally or inside each lang pack). Basically graphlib supported UTF-8 fonts without problems at all. We've updated the central TrueType font and now it supports Latin, Cyrillic and Greek without problems. Some old not-needed fonts have been deleted. And unsupported languages will need to install their language fonts (under moodledata/lang/xx_utf8/fonts).
  18. (D,C) RSS block. Working fine now with the new texlib.class.php library.
  19. (D,A) Languages list issues, not showing properly lang names. Not action required because now all those names are UTF-8 and they should work properly. I think it's an browser/font issue.
  20. (D,C) The Assingment module says that 0 words under some languages without word separators. A new configuration option has been added to count letters instead of words.
  21. (D,C) Modify the email_to_user() function to enable encoding of mails based on site setting and/or user preference. This is a must because a lot of mail clients/widgets don't support UTF-8 encodings. Done (email encoding now can be specified at site and user level plus support any charset (header + body).
  22. (N,C) Look for all the occurrences of "en" or 'en' is source code and change them to 'en_utf8' is necessary. Bug 6124
  23. (D,A) New lang edition interface for utf_8 lang packs. It should support dataroot lang packs, use 'en_utf8' as master language and "lock" the new langconfig.php file.
  24. (D,C) DB Migration with this functionalities:
    • Convert all the users/courses/site languages to their new alternative.
    • Recover from crash smoothly.
    • Handle and central tables and official modules. Contrib modules should implement their own script. See UTF-8 contrib for more info.
  25. (N,C) Modify the document_file() function to work properly (links to docs.moodle.org). If documentation isn't going to be included with Moodle anymore, such function will disappear completely in a near future.
  26. (D,A) Modify the footer output to allow it to go to the proper (en, es...) wiki page in https://docs.moodle.org. A list of available languages can be harcoded, defaulting to en.
  27. (D,C) Review the RSS feeds creator to detect if conversion to UTF-8 is needed.
  28. (D,C) Analyse if we need the $SESSION->encoding. Removed from main CVS, 3rd party plugins must be updated anyway to use curent_curset() and $CFG->unicodedb to be fully utf8 and 1.6 compatible.
  29. (D,A) Delete all the get/print_string('thisencoding') and change them to the new current_charset() function.Seems finished.
  30. (D,C) Analyse and, if possible, implement a bit improved moodle_setlocale() function because of differences between Unix locales and Win32 locales. They should go to a new string inside each langconfig.php file and, after OS detection use the correct one. Done, 95% of Windows langs will work.
  31. (D,C) Implement one PEAR download utility if finally we cannot add it to standard distro (it's ready since some time ago under http://download.moodle.org/pear). Needed to generate Excel files and, potentially, it'll grow. Finally we got perms from Xavier Noguer (see lib/pear/README.txt for a note about the PHP license and Moodle) and it has been included. Anyway, the pear download continues being generated in download.moodle.org daily.
  32. (D,C) Handling of passwords. After DB conversion, password hashes are updated to utf8 during next user login. First is tried hash of unicode plain text password, then the plain text is converted into 'oldcharset' defined in the new language pack and its hash is checked again.
  33. (D,A) Modify (and potentially upgrade) the MyPHPAdmin module in order to recognise new lang pack names. Done: just support to utf8 langs added.
  34. (N,C) POSTPONED: Add one new parameter (xxx_original_encoding) to authentication methods in order to be able to convert from external sources (ldap, db...) encoding to utf8 if Moodle is running in that mode avoiding the current utf8_decode() implemented. Bug 6123
  35. (D,A) Modify the install.php script to be able to detect DB encoding and warn about it plus use the new environment stuff to perform tests.
  36. (D,A) Create a collection of installer.php (to avoid conflicts with old install.php) files to be stored under the install/lang directory and to be used exclusively in the installation process. Hack get_string() to support this files ONLY in installation and make a script to be able to build them daily from contents existing in other language files (i.e, no manual handling of them!). Also, perhaps, add the possibility of language download at the end of the installation script.

Patches reference

Post here all the solutions you know if you consider that they'll be interesting to solve some of the pending items in the list above.


Patches to allow Japanese Language Moodles that function without garbling have been prepared and are available at at the following sites.

These patches are explained here http://moodle.org/mod/forum/discuss.php?d=13558

Mr. Kashiwagi's "Supertak" Patch (described in the thread above) http://www.supertak.com/down/sample.htm

Prof Kita's patches and rpms (based in part on Mr. Kashiwagi's) http://t-kita.net/rpm/FC/moodle/
A read me file in English describing the rpm http://t-kita.net/rpm/moodle/README-rpm-en.txt A patch describing all the things that need to be done http://t-kita.net/rpm/FC/moodle/patches/moodle-t-kita.patch Many, or even most end-users (including myself = Tim) are not sufficiently confident making extensive patches, so our Moodles have been garbling in important areas (email/excel).

One interesting PHP-UTF8 reference: http://www.phpwact.org/php/i18n/utf-8#mail_functions

Great reference for Unicode Fonts: http://www.unifont.org/fontguide/ and http://www.travelphrases.info/fonts.html

For item #8: may be for letter-index use "distinct" first letters from existing data, and not lang alphabet? it can help in multilanguage glossary... --Ne Nashev 13:53, 29 March 2006 (WST)