User Tools

  • Logged in as: anonymous (anonymous)
  • Log Out

Site Tools


mantisbt:importexport

Import Export for a list of bugs

Introduction

A complete solution for copying a list of issues between two different Mantis instances is something asked from quite some time:

For a greater flexibility, the use of an XML format looks like the best option (XML export was also asked in http://www.mantisbt.org/bugs/view.php?id=4024).

Implementation Notes

We assume a Mantis instance (A) as the source, and another instance (B) as the destination. A possible workflow could be:

  • Select a set of issues in A using filters
  • Activate the Export command
  • An XML file is produced and downloaded
  • Log in instance B as project manager
  • Import the XML file and watch instance B populated with new bugs

The first implementation will support all the issue fields (including extended info) and custom fields.

Given the extensible nature of the XML format, this feature can be enhanced later to support notes, attachments and all other informations related to the issue.

Design choices and issues

The main hurdle for this feature is to define the behaviour when any piece of information is an internal reference to instance A.

This includes:

Project IDs

Issue

the selected issues could be from more than one project in instance A and there is no guarantee the same project(s) will be in instance B

Proposed solution

import will happen on a single project in instance B

Status

DONE

User IDs

Issue

user ids will not match between the instances

Proposed solution

On import page, ask what to do with users. Options are:

  • squash all ids to a single one in instance B
  • attempt to find a matching user by username, then fall back to squashing id

Status

DONE A function implementing the above behaviour exists, but composing a list of users is memory intensive (more than 32Mb to execute with the mantis DB) so I'm refraining from using it.

The current implementation tries by default to match usernames, then squash to the userID performing the import operation

Issue

Notes and bugs can contain references to other notes or bugs (by default they are made with the syntax #bugId and ~noteId)

Proposed solution

The references to bugs and notes that are imported in the same transaction can be rewrote to reflect the new id assigned in instance B.

On import page, allow choosing whether to try converting references and one of the following default actions:

  • Replace internal links with links to original items in instance A
  • Replace them with fake links, pretending the linked bug or note was deleted

Status

DONE (notes are not in the initial implementation, but code can support those)

Categories

Issue

Category is a required field, but nothing guarantees the target project on instance B will use the same ones as instance A.

Proposed solution

Add import options:

  • skip import of issues with unknown categories
  • create missing categories (dangerous…)

Status

PENDING

Metadata

In order to support all the options above, some additional information will need to be included in the exported file.

  • Full URL ($g_path) of instance A
  • The bug link trigger ( $g_bug_link_tag )
  • The note link trigger ( $g_bugnote_link_tag )

Action Items

Phase 1

  • Prepare a basic DTD for the XML file (just bug data)
  • Prepare a sample XML file
  • Add “XML Export” link to view_all_bug_page.php
  • Add a new link in the manage menu to manage_import_issues_page.php
  • Create the manage_import_issues_page.php page

Phase 2

  • Add bug notes to the DTD

Import Issues Page details

The import strategy will be:

  • STEP 1: let the manager upload the input file
  • STEP 2: ask what to do with user IDs and bug(note) links as drafted above
  • STEP 3: perform the operation

STEP 3 details

  open XML file
  initialize oldID => newID array for: issues, notes
  for each <issue> element
    create a new Bugdata object
    populate data with <issue> fields
    save it into DB
    save oldID => newID pair

  for each element in (issues, notes)
    load data from DB
    search fields for links
    for each link found
      if link ID is known
        replace it with known newID
      else
        replace it with a conventional string or with URL to item in instance A

Requirements

The code being developed depends on:

  • PHP 5
  • XMLReader module
  • XMLWRiter module

Other

  • The resulting xml file can be tested for validity with xmllint like:
''xmllint --noout --dtdvalid mantis.dtd export_file.xml''
  • Check memory consumption when a huge number of issues are exported and imported
  • Add checks for required extensions - bool extension_loaded ( string $name )
  • let user choose destination project on instance B

Chat Logs

Mar 16 22:27:57 vboctor	giallu, I've submitted a note with my comments on the XML export.
Mar 16 22:28:30 nuclear_eclipse	hi vboctor
Mar 16 22:28:48 vboctor	howdy
Mar 16 22:35:13 giallu	back
Mar 16 22:38:17 vboctor	giallu, let me know your thoughts about the xml comments.
Mar 16 22:38:48 giallu	have some time to discuss that now?
Mar 16 22:38:55 vboctor	yes
Mar 16 22:39:17 giallu	so, let's start from the DTD
Mar 16 22:39:43 nuclear_eclipse	are you two talking about the DocBook manuals?
Mar 16 22:39:58 vboctor	no, we are discussion an import/export xml format for Mantis.
Mar 16 22:40:07 nuclear_eclipse	ah, ok, nm :P
Mar 16 22:40:16 giallu	I just remembered seeing a dtd in bugzilla, so I read some documentation about DTD and adapted it to our purposes
Mar 16 22:40:36 giallu	later I realized there exists XSD
Mar 16 22:41:11 vboctor	ok, I guess this is not a big deal.  Once we define the format we can make a DTD, XSD or both.
Mar 16 22:41:13 giallu	but I don't really want to complicate more this stuff (and I'm on a schedule becase as you probably remember this is a sponsored issue)
Mar 16 22:41:31 giallu	then
Mar 16 22:41:59 giallu	the DTD defines the fields _and_ their order
Mar 16 22:42:03 vboctor	I don't mind if we don't implement all the specified requirements.  However, whatever we implement should fall into a preset vision.
Mar 16 22:42:04 thraxisp	Are you writing import as well as export?
Mar 16 22:42:12 giallu	thraxisp, yap
Mar 16 22:42:49 thraxisp	A DTD will be a must for some one to use the import/export to interface to another system.
Mar 16 22:43:18 giallu	thraxisp, see http://www.mantisbt.org/wiki/doku.php/mantisbt:importexport for more details
Mar 16 22:43:24 vboctor	thraxisp, by DTD, you mean a DTD (or XSD) or specifically a DTD?
Mar 16 22:43:49 giallu	I don't think it makes any difference if we have a DTD or XSD 
Mar 16 22:44:14 giallu	but having a defined format is a prerequisite for writing conversion stuff
Mar 16 22:44:22 thraxisp	As I recall, a DTD defines the structure and semantics on the XML file. (XSD may do the same).
Mar 16 22:45:08 giallu	thraxisp, right. additionally AFAICT with XSD you have more powerful definition rules
Mar 16 22:45:31 giallu	like this field contains strings, integers in this range, booleans etc
Mar 16 22:47:06 vboctor	XSD and DTD both define schema.  However, XSD is in XML format and is a newer standard compared to DTD.
Mar 16 22:47:07 giallu	vboctor, so, about your comment (2), I ordered the fields in the dtd with the same ordering I got them from filter_get_bug_rows
Mar 16 22:47:50 vboctor	why do the two orders have to match?
Mar 16 22:48:03 giallu	if we want a different ordering it's an additional step we need to carry somewhere, but
Mar 16 22:48:07 vboctor	the ordering returned by filter_get_bug_rows can change any time.
Mar 16 22:49:08 vboctor	Don't you get an array of issues, then you have your logic that translates the BugData into XML?  Hence, you have to specify the order anyway?
Mar 16 22:49:27 vboctor	If you don't have this extra step, then a chance in the internal code will break the DTD right away.
Mar 16 22:49:32 giallu	I wonder wht's the purpose of having a specific ordering, or having readable dates, etc. the file's purpose is meant to be a transport between two mantis instances
Mar 16 22:49:39 vboctor	Also adding extra native fields will likely change your format as well.
Mar 16 22:50:06 vboctor	Then why not use binary format?
Mar 16 22:50:09 vboctor	or serialize?
Mar 16 22:50:29 vboctor	The aim of using XML is to have a format that people can easily use to define their own tools based on a stable format.
Mar 16 22:50:35 giallu	vboctor, anyway, yes, I can force an ordering on the fields, it's just that right now I haven't done it ;)
Mar 16 22:50:39 vboctor	this can be export from product X and import into Mantis.
Mar 16 22:51:03 vboctor	I think the schema shouldn't require a specific order.
Mar 16 22:51:19 vboctor	there is no reason why field A must show up before field B.
Mar 16 22:51:30 giallu	vboctor, the xml validates _only_ if the order is the same
Mar 16 22:51:52 giallu	so whatever we write as order in the DTD, we need to use on the XML
Mar 16 22:52:16 giallu	of course, that's just a validation issue
Mar 16 22:52:30 giallu	and the data import would be perfectly fine with any ordering
Mar 16 22:53:37 vboctor	In my opinion, the validation should ideally be a step that we execute before importing start.  This is to avoid failing mid way during import.  Specially given that we don't use transactions.  Unless we are planning to use transactions.
Mar 16 22:53:51 giallu	so, for instance, I could ksort the array from filters and be done with this
Mar 16 22:54:34 vboctor	I guess ordering is a not a big deal.
Mar 16 22:54:42 giallu	yap. let's move on
Mar 16 22:55:05 vboctor	I think the best option is to have a map which defines the name mapping / ordering, but it is not a big deal.
Mar 16 22:55:48 giallu	so let's quicly summarize your comments:
Mar 16 22:55:56 giallu	1. The export has a lot of ids. These ids may mean different things to different people working with the data. I had the same problem when defining the SOAP API interface. What I ended up doing is to use an object reference concept. Hence, rather than just using an id, I use an id and a non-localized name to refer to an object. This applies to enumerations, users, etc.
Mar 16 22:56:42 thraxisp	A map might also help for foreign import where fields are missing.
Mar 16 22:56:58 giallu	I guess I can use the same system as used in SOAP. I'll add that as needed though. is that OK?
Mar 16 22:58:47 vboctor	what do you mean by as needed?
Mar 16 22:59:06 *	nt (n=tuckern@spc1-pete6-0-0-cust391.asfd.broadband.ntl.com) has joined #mantishelp
Mar 16 22:59:08 vboctor	What I think we should do now is to have <reporter><id>1</id><name>administrator</name></reporter>
Mar 16 22:59:34 vboctor	Same for view status: <view_state><id>10</id><name>public</name></view_state>
Mar 16 23:00:11 vboctor	The other option is to use <view_state_id>10</view_state_id> and then have a section in the XML that defines all the possible view states.
Mar 16 23:00:38 giallu	it means I could assume in a first instance that we are talking about two identical installations, so states are the same
Mar 16 23:01:05 vboctor	But you can't assume that for users, projects, etc.
Mar 16 23:01:07 giallu	then we can expand to your proposal in order to support some euristic (is that the word?)
Mar 16 23:01:50 vboctor	how are you planning to handle users, projects, etc?
Mar 16 23:01:51 giallu	vboctor, no. the users are to be handled like drafted in the reqiurement page 
Mar 16 23:01:56 giallu	i.e.
Mar 16 23:02:49 *	vboctor checked the wiki page.
Mar 16 23:02:53 giallu	define a squash-to userid during import and ask admin if everything has to be squashed to that user or just the not matching ids
Mar 16 23:02:58 vboctor	your sample xml didn't include user names.
Mar 16 23:03:42 giallu	for projects. I think I'm limiting the import to a _single_ project
Mar 16 23:04:12 giallu	I don't really want to guess which project an issue belongs to
Mar 16 23:04:34 giallu	vboctor, which names?
Mar 16 23:04:58 giallu	reporters?
Mar 16 23:05:14 vboctor	I just think having <project><id>5</id><name>Mantis</name></project> will allow us to match by name even if ids don't match.
Mar 16 23:05:29 vboctor	by user names, i meant reporters, handlers, etc.
Mar 16 23:05:50 giallu	so basically, everywhere we have an id...
Mar 16 23:06:08 CIA-11	mantisbt: thraxisp * r5121 /branches/BRANCH_1_1_0/mantisbt/adm_config_report.php:
Mar 16 23:06:08 CIA-11	mantisbt: partial fix for #0008976: Remote Code Execution in adm_config
Mar 16 23:06:08 CIA-11	mantisbt:  - hide update form for those who can't change items
Mar 16 23:08:36 vboctor	giallu, yes.  I think it will make the format more flexible and we won't have to make breaking changes.
Mar 16 23:09:24 giallu	vboctor, well you know, the X in XML means we shouldn't have this problem ;)
Mar 16 23:10:38 vboctor	just end up with more code to maintain for no strong reason.
Mar 16 23:11:02 vboctor	now we need to handle <reporter_id> and <reporter>
Mar 16 23:11:34 vboctor	do you think it is a lot more work to export id/name combinations?
Mar 16 23:11:58 vboctor	or is the logic to be intelligent about such combinations on import is what is worrying you?
Mar 16 23:12:36 giallu	yeah. it's just more work in the import side (which is still a bit foggy in my head)
Mar 16 23:12:57 giallu	I think I can do that
Mar 16 23:13:06 giallu	just one more Q
Mar 16 23:13:33 giallu	do you think your 4 years old note about using DOM XML still apply?
Mar 16 23:14:02 giallu	if possible, I'd rather use some existing php module for the job
Mar 16 23:14:27 giallu	though right now I'm writing it with custom code
Mar 16 23:14:35 CIA-11	mantisbt: thraxisp * r5122 /trunk/mantisbt/adm_config_report.php:
Mar 16 23:14:35 CIA-11	mantisbt: partial fix for #0008980: Port: Remote Code Execution in adm_config
Mar 16 23:14:35 CIA-11	mantisbt:  - hide change form from unauthorized users
Mar 16 23:15:17 vboctor	custom code is fine.
Mar 16 23:16:11 giallu	yes. but I'm asking if it's fine to use some xml related functionality. The parsing is trickier
Mar 16 23:17:50 vboctor	I don't understand your question, probably since I don't remember the comment you are referring to.
Mar 16 23:17:58 *	vboctor checking...
Mar 16 23:18:13 giallu	ah. you asked to _not_ use DOM XML
Mar 16 23:18:34 giallu	becasue it was possibly not available everywhere
Mar 16 23:19:42 vboctor	yep, just read it now.
Mar 16 23:19:48 vboctor	what is the status of DOM XML and PHP 5?
Mar 16 23:19:59 vboctor	I remeber there was also Simple XML in PHP 5?
Mar 16 23:20:36 giallu	right
Mar 16 23:21:02 vboctor	Given that export / import is not a core feature, I am ok with using a PHP extension.
Mar 16 23:21:09 giallu	great
Mar 16 23:21:17 vboctor	Similar to twitter notifications that is dependent on curl extension.
Mar 16 23:21:37 giallu	I'm probably leaving the exporter as is unless I find it unmaintenable
Mar 16 23:22:21 giallu	but I'm definitely looking for some cooked solution about import
Mar 16 23:22:43 vboctor	consistency between export / import would be ideal.
Mar 16 23:23:06 vboctor	you also need to be careful that PHP doesn't run out of memory when exporting / importing a lot of issues.
Mar 16 23:23:47 giallu	ok. I'm noting this on the wiki page
Mar 16 23:33:20 vboctor	good
mantisbt/importexport.txt · Last modified: 2008/10/29 04:25 by 127.0.0.1

Driven by DokuWiki