Help... Unicode characters don't work..

[chanchao]

Hi,

This is really strange.. My site is mostly in English, but has some Thai language pages. I can add a menu item with a name in Thai just fine. I can also enter section and category names in Thai, and they show up just fine in /administrator.

However when I write an actual item with Thai characters in it, it shows up as garbled gibberish after clicking 'apply'. (Same when viewing the site).

Initially I thought it was a problem with the WYSIWYG editor so I turned that off.. However the problem remains. I'm getting worried now.

The problem is the same if I just type Thai directly, or when I use a web editor first and then save it so it converts to unicode codes.

For example, I entered " สุขสันต์ วันปีใหม " by just typing which didn't work, but also when entering the &-number codes like " สุขสันต์ วันปีใหม่ "

it doesn't work either.. (I'm using the latest stable Mambo 4.5.2)

HELP..!!!

[chanchao]

Hmm.. note that this forum handles Thai quite a bit better.. also the unicode numbers like amp#3626;amp#3640;amp#3586;amp#3626; and so on got entered correctly here..

[chanchao]

Please, anyone?? It seems to me that Mambo works in more languages than just English/Latin.. Thai language isn't even double-byte.. there's no reason why it shouldn't work! I used to have Thai language pages up on my site in plane html.. what does Mambo do that garbles it all up?

[chanchao]

Ok, SOLVED!!!

Found it a bit tricky.. First I downloaded and installed the Thai language pack. This allows setting the language to Thai and adding Thai posts, BUT it also translates all the controls of the site to Thai! That last bit was not desired because most of my site is in English, I merely wanted to add a bunch of Thai pages.

To solve that, you don't even need the Thai language pack, all you have to do is find the 'english.php' file in the 'language' subfolder in your Mambo installation. There find the line that says :

/** templates/*.php */
DEFINE('_ISO','charset=iso-8859-1');

Just change the charset to TIS-620 for Thai, so you get:

DEFINE('_ISO','charset=tis-620');

That's it! From then on you can also post in Thai language without switching first.

Now then.. all that's left to make things perfect is a swich where the visitor of the website can swtich to Thai or English depending on his/her preference... But that's for another day.

Cheers,
Chanchao

[tijs]

I have the same problem and haven't found a solution either. My site is mainly in English but also has some Hebrew, Arabic and Persian articles. The problem is that the editor (I tried all of them) converts the characters in the title to their unicode entities. In itself I could live with that, but the trouble is that because of that the title length maximum size of 100 characters is easily exceeded. In the meantime I increased this to 255 characters but that is still not enough when every character in e.g. Persian is converted to 7 characters (say, & # 1236 ; (spaces added)). It also looks very messy if you only see this Unicode gibberish in the content manager.

I tested with yesterday's 4.5.x CVS and it only got worse. There the Persian characters are transformed into something like *8'G1'* /1 .Ì'('F 'FBD' making any multilanguage website other than one with different Western languages impossible. (though I suppose this will be solved in the final 4.5.3 which is supposed to have several multi-language improvements)

Devs, any thoughts on this?

Cf. also my post Unicode character limit in title - anybody a workaround? for which I still haven't found a solution.

Chanchao, I don't think your workaround is a good one as it seems like the English articles now also have the Thai character set, which may give you problems and will certainly do so for me, since I have several different non-Western languages. Which one should I pick then?

[chanchao]

Yes, I know everything gets the Thai encoding. However I'm relatively fortunate that the whole Thai character set fits into the upper-ascii range, with the lower ascii range getting the regular English set. So English text still displayes.

But you're right that would result in more trouble for example in German or French (or Dutch to a lesser extent) where you have more use of high ascii accented characters.. English has almost none of that. Only issue perhaps would be with curly quotes, copyright signs, the Euro-symbol and so on. Those would get shown as a Thai character then. (I think.. let's test that.. )

Yup.. Just checked.. In a word like ' café ' the accented e gets displayed as a Thai character.. (Unless the visitor manually changes encoding to Latin)

Anyway, like I said I'm fortunate that the Thai set leaves the lower ascii part as standard US, AND that English uses almost no accented characters.

I read somehting about the Mambelfish project though, that seems to be a way to switch languages? Haven't installed it yet though, I think I'll just leave things as they are for now until a next Mambo version allows a user accessible control to switch things around.