UTF-8 lang files

Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

UTF-8 migration > Moving language files

Moving Language files

Some preliminary premises:

1. Moodle will run both under Unicode DBs and without them.
2. Unicode transition will be recommended intensively a well explained documentation about the DB conversion will be available. But won't be mandatory.
3. Old lang packs will be located where they are now (cvs_moodle/moodle/lang), branch 153, although they will be out from the distros.
4. New lang packs will be located under cvs_moodle/lang.
5. Lang packs (old and new) could run both from

   $dirroot->lang
   

   and

   $moodledata->lang
   

   .

The plan

1. 1.6 lang packs will be easily recognized by having one langconfig.php file where all those problematic strings will reside (thischarset, parentlang...). For now, we'll maintain it in moodle.php, until all the 1.6 code was modified. This langconfig.php file won't be editable from the lang interface to avoid some common errors in the past. It will be created by Koen when the lang pack is committed initially to CVS.
2. One new global variable (CFG->unicodedb) will be created ONLY once the DB migration was performed. Some parts of Moodle (backup/restore, mail...) will work slightly different depending upon this flag.
3. Sites with CFG->unicodedb not set, will be able to use both 1.5 and 1.6 lang packs.
4. Sites with CFG->unicodedb set, only will be able to use 1.6 lang packs.
5. The "old" CFG->unicode setting only will affect sites with CFG->unicodedb not set. By having both this two variables we'll be able to detect 1.5 sites with unicode contents (easier to migrate). The old $CFG->unicode variable is currently used to a few things, but basically to set the Content-Type header to UTF-8 (force). Also it's used to perform some specific UTF-8 string manipulation, if I'm not wrong (I cannot remember where it's). But I think it's better to have a new variable to save the migration status and, definitively, to decide a bunch of things:
   • With $CFG->unicodedb set:
     • Only 1.6 lang packs will be allowed.
     • HTTP headers will be UTF-8.
     • Specialized UTF8 functions (library/textlib.php) will be used here and there.
   • Without it:
     • Both 1.5 and 1.6 lang packs will be allowed.
     • Everything will work as it was before 1.6 (relaying in the old $CFG->unicode).
6. All the new 1.6 created sites ONLY will be created under Unicode. We'll check if it fits all the conditions before proceed with installation (DB version, encoding..). Finally we have relaxed this a bit and new sites can be created without Unicode but it have to be checked and one red warning is showed about potential problems in the future.
7. Translators will be able to:
   • Maintain the old lang packs in the MOODLE_15_STABLE branch, using Moodle 1.5. Only for old bugfixing.
   • Maintain new lang packs in HEAD, using Moodle >= 1.6. Main work must happen here, as usual. The only difference is that they will be working against lang packages stored under dataroot/lang instead of moodle/lang so they'll perform CVS operations form there.
8. ONLY the new en_utf8 lang pack (and the old en) will be included in the distro.
9. We'll maintain the new lang pack names as "xx_utf8" to avoid problems to translators but strip such suffix from the lang menu to make it transparent for the rest of users. If finally we decide to hide the pack name under 1.6, the "thislanguage" string should be a bit more self-explanatory.
10. In order to allow multilang installation, a bunch of "special" lang files will be created under install/lang. Such files, automatically maintained with one script, will contain all the strings needed in the installation process. No direct manipulation of that files is allowed.
11. A new method to download and install ONLY 1.6 lang packs must be available soon (is available in head now). I think we can start to package them in some days without publicity until the download method was stable. This download feature will be available ONLY after installation, but not before it. At install time, only the install/lang/xx/php will be available (and needed). Else, English (see point k). The download should support both install, uninstall and check for update. That download/install lang packs feature should be ONLY available for 1.6 lang packs. Main points are here:
    • They will be in a separate dir at download.moodle.org (/lang16). It's easy to create one sh script to do the work.
    • Apart from the lang packs (one zip file for each one) we'll maintain a text file with some info inside it (languages.md5):
      • Package name (en_utf8...)
      • MD5 of the zip file.
      • Lang national name ("thislangname" string).
    • So, the download utility will get that file and check it against the latest downloaded one (that will be stored in one xx_utf8.md5 file). It will show the changes and allow to download the changed packages.
    • Downloaded packages will be unzipped and their new md5 file will be created.
    • People without file(URL) access (why not to use Snoopy?) will download, copy and expand such packages under moodledata/lang manually. Obviously one new page at downloads.moodle.org should be available for them. There, they will be able to check the latest modification date for each lang pack to decide if they need to download it.
    • The automatic download and install feature only will check and work for $CFG->dataroot/lang 1.6 lang packs, and only when the new CFG->unicodedb was set.

Go to the next document: Converting the database

Known changes to implement

Changes to support 100% the moodledata/lang dir (only 1.6!).

SESSION->lang lang menu other places

Language editing: New directory only under 1.6 (forced!). Old directory 1.5 only.

Download of lang packs directly to moodledata/lang

Automatic generation of install.php lang files.

Where are we going to store all that stuff and how is Moodle distribution going to handle it?

All hardcoded ocurrences of "/en/" should be checked and transformed depending of CFG->unicodedb

One link explaining the "vendors branches" thing (it's being used to automatically create the 1.6 lang packs): http://efault.net/npat/docs_and_postings/cvs-tracking/cvs-tracking.txt

Discuss in Languages forum