Comments on: Switching to UTF-8 http://blog.punbb.org/2007/02/12/switching-to-utf-8/ Straight from the horse's mouth Thu, 28 Aug 2008 16:55:11 +0000 http://wordpress.org/?v=2.6-bleeding By: Rickard http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-25476 Rickard Wed, 05 Mar 2008 12:34:31 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-25476 Reality: Please inform us how we can commit more to UTF-8 and internationalization. I'm dying to know. Reality: Please inform us how we can commit more to UTF-8 and internationalization. I’m dying to know.

]]> By: Reality http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-25447 Reality Wed, 05 Mar 2008 03:53:42 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-25447 The truth is, you aren't as committed to UTF-8 and internationalization like you make it seem in this blog post. Especially you Jérémie. The truth is, you aren’t as committed to UTF-8 and internationalization like you make it seem in this blog post. Especially you Jérémie.

]]> By: Alex’s Blog » Encoding! eek http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-6959 Alex’s Blog » Encoding! eek Thu, 06 Sep 2007 03:09:54 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-6959 [...] Switching to UTF-8 [...] [...] Switching to UTF-8 [...]

]]> By: Asbjørn Ulsberg http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-75 Asbjørn Ulsberg Fri, 23 Feb 2007 07:45:04 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-75 I've heard good things about it. Worth a shot, perhaps? I’ve heard good things about it. Worth a shot, perhaps?

]]> By: Rickard http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-62 Rickard Thu, 22 Feb 2007 10:47:24 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-62 Yeah, maybe. I have never heard of this particular library though. Not sure how well it works. Yeah, maybe. I have never heard of this particular library though. Not sure how well it works.

]]> By: Asbjørn Ulsberg http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-58 Asbjørn Ulsberg Thu, 22 Feb 2007 10:10:22 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-58 Another option is to use the following pure PHP conversion library if iconv isn't available: http://mikolajj.republika.pl/ Another option is to use the following pure PHP conversion library if iconv isn’t available:

http://mikolajj.republika.pl/

]]> By: Rickard http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-55 Rickard Thu, 22 Feb 2007 08:31:38 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-55 I'm not sure that's such a good idea. If anything, we'll link to iconv binaries for various platforms with very clear instructions on how to use it. I’m not sure that’s such a good idea. If anything, we’ll link to iconv binaries for various platforms with very clear instructions on how to use it.

]]> By: Asbjørn Ulsberg http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-49 Asbjørn Ulsberg Wed, 21 Feb 2007 23:20:28 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-49 How about providing the iconv conversion through punbb.org somehow? You could even do it automatically and remotely by calling a PHP script on punbb.org from the db_update script. If you do it in a similar way to how the conversion script works, by splitting up the whole task in several smaller tasks, you don't have to think about timeouts either. To protect against DoS attacks on this punbb.org-hosted iconv script, you should probably do some kind of authentication plus a implement a repeat-attack protection. Nothing fancy, but it's better to be on the safe side. :) How about providing the iconv conversion through punbb.org somehow? You could even do it automatically and remotely by calling a PHP script on punbb.org from the db_update script. If you do it in a similar way to how the conversion script works, by splitting up the whole task in several smaller tasks, you don’t have to think about timeouts either. To protect against DoS attacks on this punbb.org-hosted iconv script, you should probably do some kind of authentication plus a implement a repeat-attack protection. Nothing fancy, but it’s better to be on the safe side. :)

]]> By: Rickard http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-36 Rickard Wed, 21 Feb 2007 12:01:59 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-36 Jérémie: You're probably right regarding usernames. There will be a hook somewhere in the registering process and we'll let the community come up with a plan for dealing with that. Asbjørn: I'm currently working on an extension of the db_update script that will process the database and do the conversion. It will work if the current encoding is iso-8859-1 or if the PHP extension iconv() is available (PHP5 and later) or if the PHP extension mb_string is available. If neither of these are true, we'll instruct the user to dump the database and run iconv on it manually. The conversion script will work like the search re-indexing script (e.g. process X amounts of entries, redirect, process X more etc). Jérémie: You’re probably right regarding usernames. There will be a hook somewhere in the registering process and we’ll let the community come up with a plan for dealing with that.

Asbjørn: I’m currently working on an extension of the db_update script that will process the database and do the conversion. It will work if the current encoding is iso-8859-1 or if the PHP extension iconv() is available (PHP5 and later) or if the PHP extension mb_string is available. If neither of these are true, we’ll instruct the user to dump the database and run iconv on it manually. The conversion script will work like the search re-indexing script (e.g. process X amounts of entries, redirect, process X more etc).

]]> By: Asbjørn Ulsberg http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-33 Asbjørn Ulsberg Wed, 21 Feb 2007 11:19:33 +0000 http://blog.punbb.org/2007/02/12/switching-to-utf-8/#comment-33 Excellent! I find it rather rediculous that PHP and MySQL have taken so extremely long time getting on the Unicode bandwagon. It was inevitable, but still they hesitated for years and years and even more years. And now when the need for UTF-8 is so screaming and gleaming it almost hurts, the problem with migrating to it lies in all developers using PHP and MySQL, like you PunBB developers. I don't envy you! But I think you're doing what's right and a terrific job at it too. I don't have any genious idea on how you should handle this, but what you could do is create the SQL conversion script from the database (should go pretty quick) so that the user can execute this script in his MySQL admin interface (phpMyAdmin, command line or whatever). Executing this script might take a while, but at least you'd leave the option to the user on how he wants to execute it. You may of course offer to execute it for him, but you should probably warn that it is going to take a while. Remember to set a long timeout on the PHP script that's going to do this! :-) The actual conversion process can be done by exporting all of the data to an SQL script, dropping all of the database tables, running iconv on the export to get it all to UTF-8, then importing the converted file to create all tables and columns (in UTF-8 if possible) and inserting all the data. Excellent! I find it rather rediculous that PHP and MySQL have taken so extremely long time getting on the Unicode bandwagon. It was inevitable, but still they hesitated for years and years and even more years. And now when the need for UTF-8 is so screaming and gleaming it almost hurts, the problem with migrating to it lies in all developers using PHP and MySQL, like you PunBB developers. I don’t envy you! But I think you’re doing what’s right and a terrific job at it too.

I don’t have any genious idea on how you should handle this, but what you could do is create the SQL conversion script from the database (should go pretty quick) so that the user can execute this script in his MySQL admin interface (phpMyAdmin, command line or whatever). Executing this script might take a while, but at least you’d leave the option to the user on how he wants to execute it. You may of course offer to execute it for him, but you should probably warn that it is going to take a while. Remember to set a long timeout on the PHP script that’s going to do this! :-)

The actual conversion process can be done by exporting all of the data to an SQL script, dropping all of the database tables, running iconv on the export to get it all to UTF-8, then importing the converted file to create all tables and columns (in UTF-8 if possible) and inserting all the data.

]]>