subject: character set hell posted: Sat, 14 Nov 2009 15:17:41 -0000
Problem:
A customer is pasting from a Word document into a web form that is
processed by PHP. Punctuation symbols get mashed. This may also
occur when pasting from Excel - possibly any Windows application
(depending on the symbols used).
Cause:
When people paste from Word etc, the character set used is CP-1252 -
but PHP, by default, runs as ISO-8859-1. For example, apostrophes are
encoded by Word in CP-1252 - so ASCII 146 is used for apostrophe,
instead of ASCII 39. This may cause issues for Mac users etc
(unknown) however it certainly collides with data-cleansing functions
such as sanitise() [not shown here - Ed]. CP-1252 uses high-order
ASCII (eg, values greater than 128) and these are all stripped by
sanitise().
Solution:
Place transcribe_cp1252_to_latin1() into sanitise(), before the high-
order ASCII stripper runs. This will convert high-order CP-1252
characters to low-order ISO-8859-1, thus preventing them being
stripped. See: http://uk.php.net/manual/en/function.strtr.php#80591
Notes:
1. Opera might display some zany symbols, BUT these are not
necessarily the same characters that are actually in the string! For
instance, Opera might show a euro symbol, a question mark and some
other Greek characters, instead of a quotation mark. However when the
ASCII of the string is examined, those zany symbols aren't there,
instead, there are codes like this: † The browser is seeing
those codes in the page, and interpreting them (incorrectly). [this
might be a bug in Opera 9.5 - Ed]
3. When debugging, do not use VNC. VNC apparently translates all text
copied to the clipboard to ISO-8859-1. Handy to know, but not useful
when troubleshooting character set issues.
4. When debugging, do not use Opera. Opera 9.5 apparently translates
all pasted text to NCRs. Pasting from a Word document, using Opera,
is not the same as pasting from that same document with Firefox 3 or
IE6. Firefox 3 and IE6 do not translate the pasted text, and leave
it in CP-1252.
5. sanitise() may also call htmlentities(), and htmlentities() may
convert some of the ISO-8859-1 characters back to NCRs, this is
normal, and OK. NCRs are not a problem, the problem is CP-1252
characters coming from Windows. Do not call htmlentities() before
transcribe_cp1252_to_latin1(), as some of the characters that come
out of transcribe_cp1252_to_latin1() do need to be converted to HTML
entities (such as aphostrophe).
6. In testing, it did not seem to matter whether the platform was
Windows 2000 or XP, or whether Word 2003 or OpenOffice 3 was used to
open the Word document containing the dodgy characters (apostrophe as
ASCII 146 etc), or whether IE6 or Firefox 3 was used.
7. CP-1252 is the character set Windows uses when the location is set
to Western Europe. Windows will use other character sets if the
location is set elsewhere, for instance, it will use CP-1250 in
Central Europe. I suspect this will require a new translation
function, however I don't have any customers pasting in CP-1250 yet,
so I don't have one at present...
8. In general, when troubleshooting character set issues, work with
raw data streams only. Any tool that auto-translates the data is
going to confuse things, especially if you're not aware of the auto-
translation at the time. Use some PHP such as string2ASCII() [not
shown here - Ed] to show every byte of the string, and its
corresponding ASCII value, to be sure that what you're seeing on
screen is the same as what PHP is seeing when it processes the data.
Also, seek to emulate the problem domain as much as possible. If a
customer is having problems pasting from Word 2003, use Word 2003 to
test. The more differences between the problem domain and the lab,
the more chance that some subtle translation will be done somewhere
that you do not notice. The result will be that you fix a problem
that the customer can't see, while leaving the problem the customer
can see intact - although you can't see it. You might also break
something else while you're there. Then you'll be in Character Set
Hell, just got back from there myself, I don't recommend it at all...
search: