Converting files to UTF-8

Moodle1.6

Some files, like Moodle import and export files and custom language packs or language files from third party modules need to be converted or treated as UTF-8 before they may be used with Moodle 1.6 and later.

*nix like computers (including Mac OS X)

Generally, this may be done with the iconv command on Unix, Linux or a Mac.

iconv -f original_charset -t utf-8 originalfile > newfile

see also the windows explanation - the script there is one for *nix computers, but used in a cygwin environment

Windows computers

For Windows, there are four methods of performing the conversion.

Method 1

Open the flat file in PSPad (a freeware editor) : http://www.pspad.com (some other editors people use: TextPad or NotePad++ or Crimson Editor, but there are many others. Windows built-in editors Notepad and Wordpad are often giving problems)
Click on Format, UTF-8
Save the file

Method 2

Download the Windows version of the iconv program. Download the "Complete package, except source" and run the setup program. The executable is located in the bin folder. Run from the command prompt (Start -> Run -> cmd) and follow the instructions as above.

Method 3

The conversion may also be done by using Cygwin, a Linux-like environment for Windows, and excecuting the iconv command in that environment. Here is an example of a working solution on Windows with Cygwin:

Create a text file, named ToUtf8.txt
Fill it with the following code

#!/bin/bash
FROM=iso-8859-1
TO=UTF-8
ICONV="iconv -f $FROM -t $TO"
# Convert
find ToUTF/ -type f -name "*" | while read fn; do
cp ${fn} ${fn}.bak
$ICONV < ${fn}.bak > ${fn}
rm ${fn}.bak
done

Two things should be changed for your local situation:

FROM is the originating encoding (the one your original files are in)
ToUTF is the foldername where the files that need to be converted are in. This folder may contain subfolders. Make sure you have a backup!

Start Cygwin.
With the cd foldername, cd.., ls commands, go to the folder on your windows machine where the ToUtf8.txt script and the ToUTF8 folder are in.
Execute the script by typing sh ToUtf8.txt and your files will be converted.

Method 4

The default Unicode format for Microsoft Excel and Wordpad is UTF-16. These files can be converted to UTF-8 using GNU Emacs 22.1

Open the file with Emacs
Enter the command C-x RET c utf-8 RET
You will then be asked what command you want this encoding to apply to
Enter the command C-x C-w then enter a new file name
The file you have saved will be UTF-8

Saving files directly as UTF-8

Most text editors these days can handle UTF-8, although you might have to tell them explicitly to do this when loading and saving files. (The notable exception to this is probably Notepad on Windows.)

Windows

You may save a file using Notepad (sometimes called "Editor") as UTF-8 but not with Wordpad.

Open Notepad
File - Save as -> there you see 3 fields set the last one called "encoding" to: UTF-8

Mac OS X

The built in text edit application has a 'Plain text encoding' option in the Save as... dialogue.

Linux

The standard Gnome Text Editor defaults to UTF-8 and has character set options when loading and saving.

Documentation