Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

UTF-8 scripts

From MoodleDocs

UTF-8 migration > Recoding PHP scripts

Recoding PHP scripts

Introduction

This page will show a list of well known UTF-8 related modifications to be applied to 1.6 in order to work better under Unicode. Some of the changes can be applied always while others would break compatibility with non UTF-8 sites so they have to be executed conditionally, i.e: 

     if (!empty($CFG->unicodedb)) {
         //Code to be executed in UTF8 mode
     } else {
         //Old code
     }

For each modification we'll show if it's executed always or conditionally, using the "A" and "C" abbreviation. Also, the current status will be maintained with "N"ot implemented, "W"ork in progress and "D"one. Obviously, "D" is the desired final status for all the modifications.

Finally, please, you are welcome to add new items to the list, thanks!, but do it at the end (to maintain its current numeration).

The List

  1. (D,A) Build one "check for 1.6 upgrade" utility under 1.5. Now it's present in the admin page. It's able to check for BD, PHP and PHP libraries, allowing function execution and built over the new environmentlib.php script. All the checks are defined under one simple XML file and one mechanism to update it from http://download.moodle.org has been provided (using the new componentlib.class.php library.
  2. (N, C) datalib.php to support collations under MySQL. This will allow to control language specific ordering from Moodle but 1.6.0 won't offer support for it. Instead, MySQL table/filed collation can be altered "manually" if the default unicode collation isn't enough.
  3. (D,A) textlib.class to handle all those utf-compliant functions. It's working since 1.5.3. Based (wrapper) on Typo3 text handling libraries it offers support for conversion between charsets and a buch of functions like (substr, strtoupper, strpos...).
  4. (N,C) XML import/export (scorm, ims, backup/restore, glossary, quizzes...). Under UTF-8 mode, both utf8_encode() and utf8_decode() won't be needed anymore.
  5. (W,A) Excel Export: A new excellib.class.php class wrapper has been built. It works with some PEAR libraries to be able to create UTF-16LE Excel files properly. How this mini-PEAR package will be distributed with Moodle must be decided (directly with the distro, if allowed by PHP people or separate download).
  6. (D,A) Modify every Excel generator to use the new library. This includes: grade/lib.php, course/grades.php, choice/report.php hotpot/report/default.php, quiz/report/analysis.php, quiz/report/overview.php, quiz/report/responses.ph and survey/download.php
  7. (D,A) Fixed the break_up_long_words() to work using the textlib functions.
  8. (W,A) Fixed the glossary to find properly the pivot (initial letter) under UTF8 (Skodak has some ideas here about the alphabet for each lang.
  9. (D,A) Modify the rss_title() function to support UTF8 chars.
  10. (W,A) htmlentities() to s() migration everywhere.
  11. (W,A) uses of substr, strlen, strpos, strtoupper... to use the new textlib class that offers utf8 savvy string manipulation functions.
  12. (N,A) Modify documentation to let users know how they MUST create their DB before installing Moodle 1.6 explaining all the benefits for being UTF-8 enabled sice the beginning.
  13. (D,A) Modify the central installation script to:
    • Check DB encoding, warning if unicode hasn't been detected.
    • Execute the environmental checks.
  14. (D,A) Modify the Windows32 Complete Package installation script to:
    • Force DB creation under UTF-8.
    • Execute the environmental checks.
  15. (D,A) The wiki module - this is due to the use of htmlentities() without specifying the character set. DFWiki does not have any known problems apparently.
  16. (N,A) UTF-8 national chars can't be used in paths/foldernames. This is not a Moodle problem but simply the limitations of the server. It's highly possible that this problem will persist in Moodle 1.6 as a global solution hasn't been found. With the arrival of DMS, FileManager virtualization and Repositories, it's possible that this issue will become less important...
  17. (N,A) GD support for UTF-8 strings. Perhaps it'll require some hacking + new fonts to be added (centrally or inside each lang pack).
  18. (D,C) RSS block. Working fine now with the new texlib.class.php library.
  19. (D,A) Languages list issues, not showing properly lang names. Not action required because now all those names are UTF-8 and they should work properly. I think it's an browser/font issue.
  20. (D,C) The Assingment module says that 0 words under some languages without word separators. A new configuration option has been added to count letters instead of words.
  21. (W,C) Modify the email_to_user() function to enable encoding of mails based on site setting and/or user preference. This is a must because a lot of mail clients/widgets don't support UTF-8 encodings.
  22. (N,C) Look for all the occurrences of "en" or 'en' is source code and change them to 'en_utf8' is necessary.
  23. (W,A) New lang edition interface for utf_8 lang packs. It should support dataroot lang packs, use 'en_utf8' as master language and "lock" the new langconfig.php file.
  24. (W,C) DB Migration with this functionalities:
    • Convert all the users/courses/site languages to their new alternative.
    • Able to be executed from command line.
  25. (N,C) Modify the document_file() function to work properly (links to docs.moodle.org)
  26. (N,C) Modify the footer output to allow it to go to the proper (en, es...) wiki page in https://docs.moodle.org. A list of available languages can be harcoded, defaulting to en.
  27. (N,C) Review the RSS feeds creator to detect if conversion to UTF-8 is needed.
  28. (N,C) Analyse if we need the $SESSION->encoding. Delete if not necessary.
  29. (W,A) Delete all the get/print_string('thisencoding') and change them to the new current_charset() function.
  30. (N,C) Analyse and, if possible, implement a bit improved moodle_set_locale() function because of differences between *nix locales and Win32 locales. They should go to a new string inside each langconfig.php file and, after OS detection use the correct one.
  31. (W,C) Implement one PEAR download utility if finally we cannot add it to standard distro (it's ready since some time ago under http://download.moodle.org/pear). Needed to generate Excel files and, potentially, it'll grow. Add it to credits.
  32. (N,C) Handling of passwords. After DB conversion, passwords containing non ASCII chars will be broken. So we should be able to check them again, but converted to the original encoding. If the second check returns true then we can safely the stored password hash for that user/course..... (every lang includes one 'oldcharset' string to know to what encoding we must convert the entered password).
  33. (N,C) Modify (and potentially upgrade) the MyPHPAdmin module in order to recognise new lang pack names.

Patches reference

Post here all the solutions you know if you consider that they'll be interesting to solve some of the pending items in the list above.

Patches to allow Japanese Language Moodles that function without garbling have been prepared and are available at at the following sites.

These patches are explained here http://moodle.org/mod/forum/discuss.php?d=13558

Mr. Kashiwagi's "Supertak" Patch (described in the thread above) http://www.supertak.com/down/sample.htm

Prof Kita's patches and rpms (based in part on Mr. Kashiwagi's) http://t-kita.net/rpm/FC/moodle/
A read me file in English describing the rpm http://t-kita.net/rpm/moodle/README-rpm-en.txt A patch describing all the things that need to be done http://t-kita.net/rpm/FC/moodle/patches/moodle-t-kita.patch Many, or even most end-users (including myself = Tim) are not sufficiently confident making extensive patches, so our Moodles have been garbling in important areas (email/excel).

Retrieved from "https://docs.moodle.org/dev/index.php?title=UTF-8_scripts&oldid=1895"
Powered by MediaWiki