View Issue Details

IDProjectCategoryView StatusLast Update
0007229mantisbtmigrationpublic2013-08-16 12:39
Reportercas Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
Status newResolutionopen 
Summary0007229: Merging multiple instances of Mantis
Description

Suppose you have multiple Mantis installations and you would like to combine them.
Then you are in trouble since there is no automated procedure.

This is why I developed a few scripts in order to automate this process as much as possible.

How does this work ?

  1. Ensure installations are on the same version
  2. Make a backup of your current database
  3. Make a copy of the Mantis databse tables you would like to merge
  4. Copy these tables into the original mantis database with a different prefix
    So if they are called mantis_bug_table, copy them as mexico_bug_table
  5. Copy the scripts in the root of your Mantis installation
  6. Add the text elements contained in " import_instance_txt.php" into custom_strings_inc.php
  7. Logon to Mantis with Admin rights
  8. Run program import_instance_page.php
  9. Follow instructions
  10. Verify results
  11. Remove the scripts

This also has support for the FAQ and Timetracking addon.

Additional Information

Please verify the max execution time in php.ini and the possible cgi time out value (under windows)

Tables that are being converted:

mantis_bug_file_table
mantis_bug_history_table
mantis_bug_monitor_table
mantis_bug_relationship_table
mantis_bug_table
mantis_bug_text_table
mantis_bugnote_table
mantis_bugnote_text_table

mantis_custom_field_project_table
mantis_custom_field_string_table
mantis_custom_field_table

mantis_project_category_table
mantis_project_file_table
mantis_project_hierarchy_table
mantis_project_table
mantis_project_user_list_table
mantis_project_version_table

mantis_user_table

mantis_filters_table
mantis_sponsorship_table
mantis_news_table

mantis_module_timetracking
mantis_faq_table

Tagspatch
Attached Files
import_instance.zip (17,419 bytes)
import_instance2.zip (17,994 bytes)
Import_instances.zip (17,994 bytes)
error.png (8,688 bytes)   
error.png (8,688 bytes)   
patch-2010-04-30.gz (1,177 bytes)
import_instance_20100526.patch (5,561 bytes)   
--- import_instance1.php	Wed May 26 13:01:02 2010
+++ import_instance1.php	Wed Jun 02 14:17:52 2010
@@ -38,11 +38,12 @@
 echo "<br>";
 echo "Removing primary keys/Adjusting tables" ;
 //
+
 $result = db_query("ALTER TABLE $g_cn_project_table DROP PRIMARY KEY ,ADD INDEX ( `id` ) ");
 $result = db_query("ALTER TABLE $g_cn_bug_table DROP PRIMARY KEY ,ADD INDEX ( `id` ) ");
 $result = db_query("ALTER TABLE $g_cn_project_file_table DROP PRIMARY KEY ,ADD INDEX ( `id` ) ");
 $result = db_query("ALTER TABLE $g_cn_project_version_table DROP PRIMARY KEY ,ADD INDEX ( `id` ) ");
-$result = db_query("ALTER TABLE $g_cn_category_table DROP PRIMARY KEY  ");
+$result = db_query("ALTER TABLE $g_cn_category_table CHANGE `id` `id` INT( 10 ) UNSIGNED NOT NULL, DROP PRIMARY KEY  ");
 $result = db_query("ALTER TABLE $g_cn_project_user_list_table DROP PRIMARY KEY  ");
 $result = db_query("ALTER TABLE $g_cn_sponsorship_table DROP PRIMARY KEY ,ADD INDEX ( `id` ) ");
 $result = db_query("ALTER TABLE $g_cn_news_table DROP PRIMARY KEY ,ADD INDEX ( `id` ) ");
@@ -57,6 +58,7 @@
 $result = db_query("ALTER TABLE $g_cn_bugnote_table DROP PRIMARY KEY ,ADD INDEX ( `id` ) ");
 $result = db_query("ALTER TABLE $g_cn_bugnote_text_table DROP PRIMARY KEY ,ADD INDEX ( `id` ) ");
 $result = db_query("ALTER TABLE $g_cn_bug_monitor_table DROP PRIMARY KEY  ");
+
 //
 $query = 'ALTER TABLE ' . $g_cn_project_user_list_table . ' ADD `new_id` INT( 7 )';
 db_query( $query );
@@ -106,8 +108,8 @@
 
 echo '<BR>';
 echo "Preparing/Handling custom_field conversion" ;
-$result1 = mysql_query("SELECT id,name, type, possible_values,default_value,valid_regexp, access_level_r, access_level_rw, length_min, length_max,advanced,require_report,require_update,display_report,display_update,require_resolved,display_resolved,display_closed,require_closed  from $g_cn_custom_field_table");
-while (list( $id,$name, $type, $possible_values,$default_value,$valid_regexp, $access_level_r, $access_level_rw, $length_min, $length_max,$advanced,$require_report,$require_update,$display_report,$display_update,$require_resolved,$display_resolved,$display_closed,$require_closed ) = mysql_fetch_row($result1)) {
+$result1 = mysql_query("SELECT id,name, type, possible_values,default_value,valid_regexp, access_level_r, access_level_rw, length_min, length_max,require_report,require_update,display_report,display_update,require_resolved,display_resolved,display_closed,require_closed  from $g_cn_custom_field_table");
+while (list( $id,$name, $type, $possible_values,$default_value,$valid_regexp, $access_level_r, $access_level_rw, $length_min, $length_max,$require_report,$require_update,$display_report,$display_update,$require_resolved,$display_resolved,$display_closed,$require_closed ) = mysql_fetch_row($result1)) {
 	$result2 = db_query("SELECT id from $t_mantis_custom_field_table where name='$name'");
 	$num_rows = db_num_rows($result2);
 	if ($num_rows>0) {
@@ -119,9 +121,9 @@
 	} else {
 		## this user is not available yet
 		$query = "INSERT INTO $t_mantis_custom_field_table
-				    ( name, type, possible_values,default_value,valid_regexp, access_level_r, access_level_rw, length_min, length_max,advanced,require_report,require_update,display_report,display_update,require_resolved,display_resolved,display_closed,require_closed )
+				    ( name, type, possible_values,default_value,valid_regexp, access_level_r, access_level_rw, length_min, length_max,require_report,require_update,display_report,display_update,require_resolved,display_resolved,display_closed,require_closed )
 				  VALUES
-				    ('$name', $type, '$possible_values','$default_value','$valid_regexp', $access_level_r, $access_level_rw, $length_min, $length_max,$advanced,$require_report,$require_update,$display_report,$display_update,$require_resolved,$display_resolved,$display_closed,$require_closed  )";
+				    ('$name', $type, '$possible_values','$default_value','$valid_regexp', $access_level_r, $access_level_rw, $length_min, $length_max,$require_report,$require_update,$display_report,$display_update,$require_resolved,$display_resolved,$display_closed,$require_closed  )";
 		db_query( $query );
 		$result5 = db_query("SELECT id from $t_mantis_custom_field_table where name='$name'");
 		$result6 = db_fetch_array($result5);
@@ -227,7 +229,7 @@
 	$notetxt_start=0;
 }
 //
-$result = db_query("SELECT MAX(ID) AS LAST_ID FROM $t_mantis_categhory_table");
+$result = db_query("SELECT MAX(ID) AS LAST_ID FROM $t_mantis_category_table");
 $result = db_fetch_array($result);
 $category_start = $result[LAST_ID]; 	
 if (!is_numeric($category_start)){
--- import_instance2.php	Wed May 26 17:31:36 2010
+++ import_instance2.php	Wed Jun 02 13:40:53 2010
@@ -43,7 +43,7 @@
 /////////////////////////////////////
 echo "<br>";
 echo "Preparing category_table" ;
-$result = db_query("update $g_cn_category_table set category_id=category_id+$category_start");
+$result = db_query("update $g_cn_category_table set id=id+$category_start, project_id=project_id+$project_start");
 /////////////////////////////////////
 echo "<br>";
 echo "Preparing project_version_table" ;
@@ -64,7 +64,8 @@
 		$new_id=0;
 	}
 	## update table that needs importing
-	$result4 = db_query("update $g_cn_project_user_list_table set user_id=$new_id where project_id=$id and new_id=$id1 ");	
+	$result4 = db_query("update $g_cn_project_user_list_table set user_id=$new_id where project_id=$id and new_id=$id1 ");
 }
 /////////////////////////////////////
 echo "<br>";
import_instance_20100526.patch (5,561 bytes)   

Relationships

has duplicate 0006933 closedryandesign Merging Two Mantis Installations 

Activities

cas

cas

2006-06-25 12:00

reporter   ~0013029

This version only syncs the user table, next one will have the same function for custom_fields.
In addition I am splitting multiple table conversions in parts of max 5000 records to avoid time outs.
Update expected shortly although testing is appreciated so it can be improved.

cas

cas

2006-07-03 15:52

reporter   ~0013062

Version 2 have been added, hope this can be used by others also.

vboctor

vboctor

2006-07-29 02:01

manager   ~0013164

Haven't looked at the code, but do you handle refernces to issues and issue notes from withing the summary, description, additional information, steps to reproduce and notes?

cas

cas

2006-07-30 07:02

reporter   ~0013170

At current they are not being "translated" since there was little use for it in our environment.
This however can be added to the code if there is enough need for it. This could be made an option since it can prove to be very time consuming on huge databases.

vboctor

vboctor

2006-10-26 03:53

manager   ~0013646

cas, it would be great if you can re-add the attachment since it was lost during the server crash.

cas

cas

2006-10-28 08:40

reporter   ~0013654

Latest version is available here again. I have emailed the scripts to various people, if they have comments add-ons, improvements, it should be reported here.

grangeway

grangeway

2008-07-13 10:14

reporter   ~0018421

Hello,

I've added the tag patch for someone to look at this attachment.

Giallu has been looking at some import/export routines, so I dont know if this would be helpful for him.

grangeway

grangeway

2008-07-15 14:10

reporter   ~0018554

Assigning to giallu to review

dapozzom

dapozzom

2008-11-04 12:40

reporter   ~0019771

I'm using 1.1.1 and trying to merg two instances. The migration process is working till step 7 when I receive the error attached. The source instance for bug_text_table has ids from 2 to 1334 the destination one from 1787 to 2281.
Do you have any ideas where the problem can be.
Thanks and regards,
Marco

cas

cas

2008-11-08 12:49

reporter   ~0019814

Will look into it when back home. Suppose you cannot share the databases so I can test with your data?
If you can, send me a pm and I will give you my email address.
Otherwise it will proof very difficult since I have done this process multiple times be it not with 1.1.1

cas

cas

2009-01-20 06:05

reporter   ~0020659

Issue was solved. It appeared that both instances were not on the saem version.

giallu

giallu

2009-01-20 07:11

reporter   ~0020661

in the latest release we made (1.2.0a3) there is a new plugin for exporting data in xml format and importing it back.

Please have a look at it, I think it makes up a good base and it should be easy to expand the coverage of the import/export routine. Feel free to ping me if you would like to help.

cas

cas

2009-01-20 13:27

reporter   ~0020665

Had already seen that, not yet time to explore.
Will check it out.

joshribakoff

joshribakoff

2009-10-31 00:38

reporter   ~0023506

This would be useful

johmart

johmart

2010-04-30 02:12

reporter   ~0025353

Last edited: 2010-04-30 02:12

Thanks a lot for this wonderful work. It saved me a lot of time!

The attached patch and new version fix the following things:

  • error in table name on executing import_instance3.php
  • error when importing bugnote texts (using wrong id)

Also introduces new feature:

  • possibility to set minimum bug number to use for imported problem reports (rather than auto-detected one)
rini17

rini17

2010-05-25 12:00

reporter   ~0025609

I was under impression that 2010-04-30 version is compatible with 1.2.x, but seems it isn't? I got error in import_instance1.php under mantis 1.2.1:

Database query failed. Error received from database was 0001146: Table 'test.kavo_project_category_table' doesn't exist for the query: ALTER TABLE kavo_project_category_table DROP PRIMARY KEY .

Problem is caused by mantis_project_category_table was renamed to mantis_category_table .

Second question, why do you drop primary keys of source tables? It makes debugging harder (I have to re-import them every time).

johmart

johmart

2010-05-26 01:00

reporter   ~0025611

The 2010-04-30 version worked for me with mantis 1.2.1.

cas

cas

2010-05-26 11:49

reporter   ~0025613

Last edited: 2010-05-26 13:25

Have reviewed the code and made some changes. There still was an issue with the category table which should be fine now.
New code uploaded for version 1.2.X

Support for FAQ & Timetracking is restricted to the plugins available.

rini17

rini17

2010-06-02 06:05

reporter   ~0025659

Now I am getting this: "0001075: Incorrect table definition; there can be only one auto column and it must be defined as a key : ALTER TABLE kavo_category_table DROP PRIMARY KEY".

I am using mysql 5.0.51a-24+lenny2+spu1 and can't upgrade. Found helpful comment at http://dev.mysql.com/doc/refman/5.0/en/alter-table.html , will try to implement it:

Attempting to "ALTER TABLE ... DROP PRIMARY KEY" on a table when an
AUTO_INCREMENT column exists in the key generates an error:

ERROR 1075 (42000): Incorrect table definition; there can be only one auto
column and it must be defined as a key.
To make this work without erroring, drop and re-add the new primary key in a
single statement, e.g.:

ALTER TABLE mytable DROP PRIMARY KEY, ADD PRIMARY KEY(col1,col2);

rini17

rini17

2010-06-02 08:42

reporter   ~0025660

Last edited: 2010-06-02 08:42

I have attached patch against version 20100526 for issues I encountered under mysql-5.0.51a-24+lenny2+spu1 and mantis 1.2.1:

  1. Above error "0001075: Incorrect table definition;....".
    --> fixed by resetting id column to non-autoincrement before dropping primary key
  2. still broken category table support
    --> fixed typo and updated columns
  3. broken custom_field table support
    --> removed all references to mantis_custom_field_table.advanced column

I have yet to thoroughly check the import results, so there could be more problems ahead. Despite everything I am still very thankful to you, cas, for your work!

derril

derril

2012-05-03 02:48

reporter   ~0031744

I found that the script written by cas didn't work for, so I've written another script to do the migration as I needed and I thought it might be useful: https://gist.github.com/2573767
I've written it very procedurally so that it's easy to understand. Feel free to add or make changes. It won't work for everyone as it's meant to deal with 2 mantis installs that have the same users and also keep the bug IDs.

Dentxinho

Dentxinho

2012-05-03 10:53

reporter   ~0031752

How to keep the same bug IDs, if you can have the same ID on both instances?

derril

derril

2012-05-03 12:53

reporter   ~0031755

I was lucky in my case that the mantis i wanted to merge into hadn't been used a lot and in the other one the project's bug IDs started after the IDs in the i wanted to merge into.
Maybe useful in only a small number of cases, but thought I make it available.

jgeorge

jgeorge

2012-08-16 09:17

reporter   ~0032580

Hi,
I have an issue related with Mantis and SQL server.
Yesterday i downloaded "mantisbt-1.2.11" and installed.
We have a running mantis pointed to a Database 'bugtracker', i don't know version of this. We forced to install because, 'Reset password' was not working in old version.

Now, i want to connect to existing database with New "mantisbt-1.2.11".
any chance?
Pls advice
thanks

RobertMartin

RobertMartin

2013-02-14 10:22

reporter   ~0035140

I've encounter an issue in step 1 of this tool, when a category name has a simple quote in it. I've solved this issue with this modification :

--- /root/res/pkg/tmp/import_instance1.php 2013-02-08 17:03:54.000000000 +0100
+++ import_instance1.php 2013-02-14 16:09:37.000000000 +0100
@@ -108,7 +108,8 @@
echo "Preparing/Handling custom_field conversion" ;
$result1 = mysql_query("SELECT id,name, type, possible_values,default_value,valid_regexp, access_level_r, access_level_rw, length_min, length_max,require_report,require_update,display_report,display_update,require_resolved,display_resolved,display_closed,require_closed from $g_cn_custom_field_table");
while (list( $id,$name, $type, $possible_values,$default_value,$valid_regexp, $access_level_r, $access_level_rw, $length_min, $length_max,$require_report,$require_update,$display_report,$display_update,$require_resolved,$display_resolved,$display_closed,$require_closed ) = mysql_fetch_row($result1)) {

  • $result2 = db_query("SELECT id from $t_mantis_custom_field_table where name='$name'");
  • $name = str_replace('\'', '\\'', $name);
  • $result2 = db_query("SELECT id from $t_mantis_custom_field_table where name='$esc_name'");
    $num_rows = db_num_rows($result2);
    if ($num_rows>0) {

    user already exists, retrieve future id

RobertMartin

RobertMartin

2013-02-14 10:59

reporter   ~0035142

I've also had this problem when trying to list bugs after the merge:

APPLICATION ERROR 0001502

Category not found.

I've solved it by applying this modification:

--- import_instance5.php 2013-02-14 16:41:35.000000000 +0100
+++ import_instance5.new.php 2013-02-14 16:35:57.000000000 +0100
@@ -28,7 +28,7 @@
/////////////////////////////////////
echo "
";
echo "Preparing bug_table" ;
-$result = db_query("update $g_cn_bug_table set id=id+$bug_start, bug_text_id=bug_text_id+$bug_start, project_id=project_id+$project_start");
+$result = db_query("update $g_cn_bug_table set id=id+$bug_start, bug_text_id=bug_text_id+$bug_start, project_id=project_id+$project_start, category_id=category_id+$category_start");
$result = db_query("update $g_cn_bug_table set duplicate_id=duplicate_id+$project_start where duplicate_id!=0");

next update this file with the correct userid

$result1 = mysql_query("SELECT id,reporter_id,handler_id from $g_cn_bug_table");
@@ -72,4 +72,4 @@
</div>

<?php
-html_page_bottom1( FILE );
\ Pas de fin de ligne à la fin du fichier.
+html_page_bottom1( FILE );

RobertMartin

RobertMartin

2013-02-14 11:42

reporter   ~0035144

I've attached a patch gathering the corrections mentionned in activity #35140 and #35142.

joshribakoff

joshribakoff

2013-03-19 13:54

reporter   ~0035916

Instead of trying to merge at the table level wouldn't it be better to add the ability to export all data to a CSV? This would be good because it would allow people to switch over from other ticket systems by importing data from a 3rd party ticket system. It would also solve the problem of merging.

The way you're going about it now by merging at the table level will make it difficult to maintain the software, every-time you change the database you'll need to update this merge feature. Its also not a standard feature. Import/export to CSV on the other hand is a very standard feature, accomplishes the same goals, and more.

cas

cas

2013-03-19 14:49

reporter   ~0035917

CSV im/ex-ports are indeed quite common but the databases of the various systems are not. It is easy enough to export all tables to csv but it will not help very much. When merging, certain logic needs to be applied, that is what it is all about.
So both serve their own purpose.

joshribakoff

joshribakoff

2013-03-20 08:21

reporter   ~0035926

Not so sure about that. When you import, you're merging new data into the system.

cas

cas

2013-03-20 10:12

reporter   ~0035927

That is correct but when importing you are simply adding issues to a (new/other) system. Merging is taking all available data (like history etcetera) from 2 installations of the same product into account.
For me, simple import was just not enough hence the development of the merging scripts/plugin.

RobertMartin

RobertMartin

2013-04-02 06:11

reporter   ~0036010

I've turned the original script to a plugin (MantisImportInstance.tgz).

My two mantis instances are now only one thanks to it.

Afterward, I realized that it shouldn't be a plugin but more a script available in the admin directory.

atrol

atrol

2013-08-16 12:39

developer   ~0037880

Removed assignment. giallu will not contribute to this issue in near future.