UTF-8 and BOM

Jump to: navigation, search

Note: You are currently viewing documentation for Moodle 2.0. Up-to-date documentation for the latest stable version is available here: UTF-8 and BOM.

Database Activity

With the Database Activity there still seems to be a problem importing UTF-8 files with BOM (http://en.wikipedia.org/wiki/Byte-order_mark).

--Frank Ralf 10:36, 13 July 2009 (UTC)

What does BOM mean?

What is it good for?

It is used for multibyte characters to mark the order in which the bytes appear.

What's the problem with the BOM?

  • See Display problems caused by the UTF-8 BOM
  • Some text editors add a BOM by default, for example Windows' Notepad.
  • When exporting from OpenOffice Calc the BOM sneaks in even after the first delimiter!

How can I detect a BOM?

You will need a text editor which is capable of showing special Unicode characters. A good Unicode text editor for Windows is SC UniPad.

Calc-export BOM.png

The picture shows an exported CSV file from OpenOffice Calc where the BOM (#FEFF) sneaks in even after the first delimiter!

Scanning Moodle folder for BOM files

If Moodle is installed on a linux server you can try one of these command lines:

find . -type f -print0 | xargs -0r awk '/^\xEF\xBB\xBF/ {print FILENAME}{nextfile}'
fgrep -rl `echo -ne '\xef\xbb\xbf'`

find & remove!

find . -type f -exec sed 's/^\xEF\xBB\xBF//' -i.bak {} \; -exec rm {}.bak \;

How can I get rid of the BOM?

Any of the above mentioned Unicode capable text editors will allow you to remove a BOM, some even automatically when opening or saving a file.

Some other text editors will save files without BOM, e.g. Notepad++.

See also