View Issue Details

IDProjectCategoryView StatusLast Update
0007779mantisbtlocalizationpublic2010-03-20 20:31
ReporterAkiacat Assigned Tograngeway  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionno change required 
Product Version1.1.0a2 
Summary0007779: [all lang] no upgrade path to 1.10 for non-latin characters
Description

Umlauts are not correctly converted into UTF-8 characters during an upgrade from 1.0.4 (possibly all 1.0.x versions are affected) to 1.1.0a2. The result is non-readable text/scrambled umlauts for users after the upgrade.

... The same for all other languages - help with upgrade code is appreciated.

TagsNo tags attached.

Relationships

related to 0006782 closedvboctor MantisBT should do "SET NAMES $charset" on connect to database 

Activities

Akiacat

Akiacat

2007-02-22 03:12

reporter   ~0014083

This can be temporarily fixed during an upgrade by the following steps:
(from http://sourceforge.net/mailarchive/message.php?msg_id=38196360)

  • After download and install iconv, then I created a new db:
    CREATE DATABASE bugtracker DEFAULT CHARACTER SET utf8

  • converted the file contents using iconv:
    iconv -f latin1 -t utf-8 backup_latin1.sql > backup_utf8.sql

  • and imported the contents of the converted file into the new database:
    mysql -uUser -pPassword bugtracker < backup_utf8.sql

Martin Fuchs

Martin Fuchs

2007-07-18 15:48

reporter   ~0015096

I think you should set the target version "1.1.0" for this entry to remember fixing the upgrade procedure before shipping the first official 1.1.x release.

Martin Fuchs

Martin Fuchs

2007-07-19 01:48

reporter   ~0015100

There is a simpler way to convert the database encoding using the mysql tools, you don't need 'iconv'. Just set the character set to UTF-8 when dumping the previous database using:

mysql -uUSER -pPASSWORD INSTANCE --default-character-set=utf8 > mantis-db.sql

And you will get a dump file encoded in UTF-8. After setting up the new database with default character set utf8 you can import this sql script to create the new database schema.

But the main problem I had after upgrading to Mantis 1.1.0a3 was that decribed in 0006782: The PHP database connection has to be changed using the statement "set names utf8" into utf8 mode in order to correctly transfer all strings from/to the database.

vboctor

vboctor

2007-07-20 03:03

manager   ~0015108

Thanks Akiacat and Martin for sharing your upgrade tips. This is really helpful. Sometime ago I've created a Wiki page that includes some hints from a message sent on the mailing list. Maybe you guys can help with editing it to include your tips.

http://www.mantisbt.org/wiki/doku.php/mantisbt:upgrade_to_utf8

Martin Fuchs

Martin Fuchs

2007-07-20 18:13

reporter   ~0015118

OK, I included the conversion examples from here under the "expert users" section into the Wiki page.

Martin Fuchs

Martin Fuchs

2007-09-03 10:39

reporter   ~0015565

Let's describe the situation before an pending upgrade:
The user is already using Mantis with an older release with a MySQL
database. There might already be used UTF8-encoding in the database
schema, for the default connection mode and/or through the Mantis
string settings (Mantis 1.1.x pre-releases). So depending on the
concrete case the encoding of the strings in the database has to be
changed in order to correctly display them in the new Mantis version.
It all comes down to three main cases:

1.) There is no need to change anything in the string encoding. This might
be the case if only latin characters (ASCII code < 127) are used in the
current locale and no special characters are used. Or there might already
been used UTF8-encoding with a Mantis 1.0.x installation.

2.) Strings have been previously stored using another encoding like e.g.
latin1 and there have been used special characters like umlauts. In this
case the strings should be re-encoded using UTF8.

3.) If one of the previous 1.1.x-alpha versions has been used to setup
and fill the database, strings might be double UTF8-encoded. To fix this,
one level of UTF8-encoding has to be reversed.

So all we have to do is support the upgrading user with an utility to handle
case 2.) and 3.). The user should decide himself which way to go looking
at his own test data. The test data should contain the special characters
which are used in the particular Mantis instance. This is also in the
responsibility of the upgrading user. So we should describe the process
and the and the why-and-how on the new administration pages.
You see, this is a try-and-error strategy. So we also need to give the user
the opportunity to go back at any time.

Now to my proposal for the new upgrade utility:
We should extent the existing upgrade page admin/upgrade_warning.php .
Instead of only warning and printing out the description of how to backup
the database we should extend the functionality and really launch the mysql
utilities from the PHP code.
The following four menu entries should provide all that's needed:

A.) Backup the database
Mantis already knows the database connection parameters, so it should
be no problem to execute a command like:
mysqldump -u[username] -p[password] [database_name] > [filename]
It should be possible to enter the filename using an input control. By
default it could be something like "/tmp/mantis-backup.sql". However it is
important to print out the error messages, which might be generated by the
dump utility in order to see if the backup was successful.

B.) Restore the database
This is similar to the Backup menu entry and should execute the following
command:
mysql -u[username] -p[password] [database_name] < [filename]
The file name should also be entered using an input control. The default
should be the same name, which was used in the Backup step. Again, printing
out the resulting error messages is important.

C.) Encode to UTF-8
To re-encode the strings in the database we can use a two step approach:
First dump the database content using this command:
mysqldump -u[username] -p[password]
[database_name] --default-character-set=utf8 > /tmp/mantis-export.sql
Then re-import the data using the default encoding with:
mysql -u[username] -p[password] [database_name] < /tmp/mantis-export.sql
The temporary file can be deleted after the successful import.
Optionally we could allow to specify another character set using an option
like "--default-character-set=latin1" instead of the default character set.

Before actually executing the command we should print out a warning message:
"Some very long strings in the database could be truncated on
conversion to UTF8."

D.) Encode from UTF-8
To remove UTF-8 encoding from the strings in the database we can go the
reverse way:
First dump the database content using this command:
mysqldump -u[username] -p[password] [database_name] >
/tmp/mantis-export.sql

Optionally we could allow to specify another character set for the dump file
using an option like "--default-character-set=latin1" instead of the default
character set.
Then re-import the data using UTF-8 encoding with:
mysql -u[username] -p[password]
[database_name] --default-character-set=utf8 < /tmp/mantis-export.sql
The temporary file can be deleted after the successful import.

siebrand

siebrand

2009-06-11 12:07

reporter   ~0022093

Paul, will your work on the database model (conversion?) be of any help here, or must the manual procedure be followed? Please take the bug, or close it, because manual steps have been described.

siebrand

siebrand

2010-03-20 20:31

reporter   ~0024831

Yes it takes some manual steps, but it is possible. So this is not considered a bug.