UTF-8 solution

2008-04-05T23:00:24Z

Pqa: /* The solution */

[[UTF-8 migration]] > [[UTF-8 justification]] > The solution

==The solution==

Now that we have explained all the problems with the schema used until now, and have viewed the benefits (immediate and future) of the migration to UTF-8, in this page we are going to organize the whole process of the conversion. Every required step will be defined briefly (you will be able to know more about each one in next chapters). Here it is:

* '''Documenting and discussing plans''': All the stuff you have read up to now, and in the rest of this Wiki page, plus active discussions in the [http://moodle.org/mod/forum/view.php?id=55 Developer forum] will be the source to understand the whole process. Aren't these enough?

* '''[[UTF-8 info gathering|Gathering detailed information from the community]]''': To have a real perception of how the UTF utilities are going to work under different configurations, one simple test has been created. By running it against as many Moodle servers as possible we will be able to detect potential problems before release.

* '''Stabilising min. requirements for Moodle 1.6''': With the information gathered in the previous step, and based in the LAMP platform evolution, current Moodle requirements have to change. Not only the UTF-8 migration, but everything will benefit with the use of new versions of LAMP. Details are available in the next page.

* '''Creating a common text library to be used everywhere''': A lot of PHP functions are not UTF-8 safe and we must provide Moodle with their UTF-8 compliant alternatives. All these specialised functions will be grouped in a new library to maintain it easier. More information in the next page.

* '''[[UTF-8 lang files|Moving language files]]''': Every language pack needs to be converted to UTF-8 plus other adjustments to run smoothly under Moodle 1.6. To maintain everything working and trying to make things easier to translators (isolating them from technical and complex CVS operations) there is a plan. Just read it, and if you want, discuss it too in the [http://moodle.org/mod/forum/view.php?f=26 Languages forum].

* [[UTF-8 DB|DB migration]]: All the contents stored at Database level must be converted, from their original charset to UTF-8. This conversion can be performed following different paths, depending of the type of site (mono or multilang), its original charset and the DB backend (MySQL or PostgreSQL).

* [[UTF-8 scripts|Recoding PHP scripts]]: Moodle code itself needs to be modified to use new string manipulation functions and to work properly under an UTF-8 environment. A list of changes and their level of implementation. The [http://moodle.org/mod/forum/view.php?id=55 Developer forum] is, once again, a good place to discuss all this stuff.

Now, let's see the [[UFT-8 initial steps|initial steps]] of the process.

[[Category:UTF-8]]

UTF-8 current

2008-04-05T22:48:50Z

Pqa: /* Current situation */

[[UTF-8 migration]] > [[UTF-8 justification]] > Current situation

==Current situation==

Up to series 1.5.x Moodle has been able to run under a lot of different languages. Each of them is provided in the form of a "Language Pack" where all the strings and help files are translated properly.

Each of these languages has the ability to define the character encoding that is going to be used when any page is sent to the browser. Furthermore such encoding is used when the user submits any form info back to Moodle. Finally, such info in manipulated by Moodle and sent to the database backend.

As languages can be determined and/or forced by Moodle at user level, course level and site level, the application knows, at any time, the encoding it has to stamp in the web page sent to the browser.

Although this schema is working relatively fine since some time ago, some important limitations are currently present and they need to be solved:

* '''Multilingual Sites''' (or, more properly, multiencoding sites): As we have seen above, each web page have one - only one! - character encoding. As traditional encodings were "exclusive" because they didn't support all the characters in the world, it's absolutely impossible to mix contents from different encodings unless we use one universal encoding. UTF-8 for example.

* '''PHP's Difficulties''': PHP has offered, in the past, a good support for different encodings, specially for those that are using only one byte per character. Support for multibyte encodings was less than perfect and some commonly used functions don't work against those encodings. Trying to add support for all the currently used encodings could be really difficult because each one has its own characteristics. So, once more, we need to go to only one multibyte encoding. UTF-8 can be it.

* '''Lack of Collation''': One important characteristic of every language is their collation rules. Basically such rules define how to sort alphabetically different strings. And those rules go from the simplest ASCII ordering to complex norms where uppercase letters are different than lowercase (in terms of graphemes or sorting) or a collection of characters have a specific place when they are together. All these strategies must be supported from Database and recent versions of both MySQL and PostgreSQL have some powerful features around this. More yet, the UTF-8 encoding defines some universal collations that can de used by default everywhere and we have the possibility to be more precise at DB level using some specialized collations too.

* '''Exchange of Information''': Moodle has a lot of features related to this. From loading users and enrolments, the import and export of glossaries, pools of questions... to the backup and restore of courses functionality, a lot of files are loaded and saved when working with Moodle. Generating all those files under different encodings can be really hard to maintain and, worse, will break compatibility between files generated under exclusive encodings. Again, we need to homogenize how all those files are handled. The solution, UTF-8, yep!

* '''Adoption of Standards''': With the arrive of a plethora of new standards (XHTML, Scorm, IMS Content Package, IMS Learning Design, SOAP, XML-RPC...) a new way to work is becoming more and more important every day: [http://en.wikipedia.org/wiki/XML XML] files. All the modern specifications about storing and sending information use this markup language internally. And, although other are supported, one encoding is the king of this documents. Do you guess which one? Of course, UTF-8, once more.

All these problem have been more obvious for people using encodings different from the ISO-8859-1 that is shared by a lot of Latin-based languages (English, French, Spanish, German, Italian...), mainly because both the web technology used (browsers), the programming environment (PHP) and the database backend (MySQL, PostgreSQL...) had a very limited support for the rest of encodings.

But nowadays, things have changed a lot. From browsers supporting different encodings to both PHP (with alternatives like [http://uk2.php.net/mbstring mbstring] or [http://uk.php.net/iconv iconv]) and Database improvements, make it practicable to tackle the problems above with high possibilities of success.

And, here, the proposed [[UTF-8 solution|solution]].

[[Category:UTF-8]]

MoodleDocs - Wkład użytkownika [pl]

UTF-8 solution

UTF-8 current