This is an old revision of the document!
Table of Contents
Import Export for a list of bugs
- Author: Gianluca Sforna (giallu)
- Status:
Draft| Work in Progress |Implemented - Associated Issue: http://www.mantisbt.org/bugs/view.php?id=4220
Introduction
A complete solution for copying a list of issues between two different Mantis instances is something asked from quite some time:
http://www.mantisbt.org/bugs/view.php?id=4220 http://www.mantisbt.org/bugs/view.php?id=5310
For a greater flexibility, the use of an XML format looks like the best option (XML export was also asked in http://www.mantisbt.org/bugs/view.php?id=4024).
Implementation Notes
We assume a Mantis instance (A) as the source, and another instance (B) as the destination. A possible workflow could be:
- Select a set of issues in A using filters
- Activate the Export command
- An XML file is produced and downloaded
- Log in instance B as project manager
- Import the XML file and watch instance B populated with new bugs
The first implementation will support all the issue fields (including extended info) and custom fields.
Given the extensible nature of the XML format, this feature can be enhanced later to support notes, attachments and all other informations related to the issue.
Design choices and issues
The main hurdle for this feature is to define the behavior when any piece of information is an internal reference to instance A.
This includes:
Project IDs
Issue
the selected issues could be from more than one project in instance A and there is no guarantee the same project(s) will be in instance B
Proposed solution
import will happen on a single project in instance B, so warn the user on export if “All project” is selected
User IDs
Issue
user ids will not match between the instances
Proposed solution
On import page, ask what to do with users. Options are:
- squash all ids to a single one in instance B
- attempt to find a matching user by username, then fallback to squashing id
Bug and Notes internal links
Issue
Notes and bugs can contain references to other notes or bugs (by default they are made with the syntax #bugId and ~noteId)
Proposed solution
The references to bugs and notes that are imported in the same transaction can be rewrote to reflect the new id assigned in instance B.
On import page, allow choosing whether to try converting references and one of the following default actions:
- Replace internal links with links to original items in instance A
- Replace them with fake links, pretending the linked bug or note was deleted
Metadata
In order to support all the options above, some additional information will need to be included in the exported file.
- Full URL ($g_path) of instance A
- The bug link trigger ( $g_bug_link_tag )
- The note link trigger ( $g_bugnote_link_tag )
Action Items
Phase 1
- Prepare a basic DTD for the XML file (just bug data)
- Prepare a sample XML file
- Add “XML Export” link to view_all_bug_page.php
- Add a new link in the manage menu to manage_import_issues_page.php
- Create the manage_import_issues_page.php page
Phase 2
- Add bug notes to the DTD
Import Issues Page details
The import strategy will be:
- STEP 1: let the manager upload the input file
- STEP 2: ask what to do with user IDs and bug(note) links as drafted above
- STEP 3: perform the operation
STEP 3 details
open XML file initialize oldID => newID array for: issues, notes for each <issue> element create a new Bugdata object populate data with <issue> fields save it into DB save oldID => newID pair for each element in (issues, notes) load data from DB search fields for links for each link found if link ID is known replace it with known newID else replace it with a conventional string or with URL to item in instance A
Requirements
The code being developed depends on:
- PHP 5
- XMLReader module
- XMLWRiter module
Other
- The resulting xml file can be tested for validity with xmllint like:
''xmllint --noout --dtdvalid mantis.dtd export_file.xml''
- Check memory consumption when a huge number of issues are exported and imported
Chat Logs
Mar 16 22:27:57 vboctor giallu, I've submitted a note with my comments on the XML export. Mar 16 22:28:30 nuclear_eclipse hi vboctor Mar 16 22:28:48 vboctor howdy Mar 16 22:35:13 giallu back Mar 16 22:38:17 vboctor giallu, let me know your thoughts about the xml comments. Mar 16 22:38:48 giallu have some time to discuss that now? Mar 16 22:38:55 vboctor yes Mar 16 22:39:17 giallu so, let's start from the DTD Mar 16 22:39:43 nuclear_eclipse are you two talking about the DocBook manuals? Mar 16 22:39:58 vboctor no, we are discussion an import/export xml format for Mantis. Mar 16 22:40:07 nuclear_eclipse ah, ok, nm :P Mar 16 22:40:16 giallu I just remembered seeing a dtd in bugzilla, so I read some documentation about DTD and adapted it to our purposes Mar 16 22:40:36 giallu later I realized there exists XSD Mar 16 22:41:11 vboctor ok, I guess this is not a big deal. Once we define the format we can make a DTD, XSD or both. Mar 16 22:41:13 giallu but I don't really want to complicate more this stuff (and I'm on a schedule becase as you probably remember this is a sponsored issue) Mar 16 22:41:31 giallu then Mar 16 22:41:59 giallu the DTD defines the fields _and_ their order Mar 16 22:42:03 vboctor I don't mind if we don't implement all the specified requirements. However, whatever we implement should fall into a preset vision. Mar 16 22:42:04 thraxisp Are you writing import as well as export? Mar 16 22:42:12 giallu thraxisp, yap Mar 16 22:42:49 thraxisp A DTD will be a must for some one to use the import/export to interface to another system. Mar 16 22:43:18 giallu thraxisp, see http://www.mantisbt.org/wiki/doku.php/mantisbt:importexport for more details Mar 16 22:43:24 vboctor thraxisp, by DTD, you mean a DTD (or XSD) or specifically a DTD? Mar 16 22:43:49 giallu I don't think it makes any difference if we have a DTD or XSD Mar 16 22:44:14 giallu but having a defined format is a prerequisite for writing conversion stuff Mar 16 22:44:22 thraxisp As I recall, a DTD defines the structure and semantics on the XML file. (XSD may do the same). Mar 16 22:45:08 giallu thraxisp, right. additionally AFAICT with XSD you have more powerful definition rules Mar 16 22:45:31 giallu like this field contains strings, integers in this range, booleans etc Mar 16 22:47:06 vboctor XSD and DTD both define schema. However, XSD is in XML format and is a newer standard compared to DTD. Mar 16 22:47:07 giallu vboctor, so, about your comment (2), I ordered the fields in the dtd with the same ordering I got them from filter_get_bug_rows Mar 16 22:47:50 vboctor why do the two orders have to match? Mar 16 22:48:03 giallu if we want a different ordering it's an additional step we need to carry somewhere, but Mar 16 22:48:07 vboctor the ordering returned by filter_get_bug_rows can change any time. Mar 16 22:49:08 vboctor Don't you get an array of issues, then you have your logic that translates the BugData into XML? Hence, you have to specify the order anyway? Mar 16 22:49:27 vboctor If you don't have this extra step, then a chance in the internal code will break the DTD right away. Mar 16 22:49:32 giallu I wonder wht's the purpose of having a specific ordering, or having readable dates, etc. the file's purpose is meant to be a transport between two mantis instances Mar 16 22:49:39 vboctor Also adding extra native fields will likely change your format as well. Mar 16 22:50:06 vboctor Then why not use binary format? Mar 16 22:50:09 vboctor or serialize? Mar 16 22:50:29 vboctor The aim of using XML is to have a format that people can easily use to define their own tools based on a stable format. Mar 16 22:50:35 giallu vboctor, anyway, yes, I can force an ordering on the fields, it's just that right now I haven't done it ;) Mar 16 22:50:39 vboctor this can be export from product X and import into Mantis. Mar 16 22:51:03 vboctor I think the schema shouldn't require a specific order. Mar 16 22:51:19 vboctor there is no reason why field A must show up before field B. Mar 16 22:51:30 giallu vboctor, the xml validates _only_ if the order is the same Mar 16 22:51:52 giallu so whatever we write as order in the DTD, we need to use on the XML Mar 16 22:52:16 giallu of course, that's just a validation issue Mar 16 22:52:30 giallu and the data import would be perfectly fine with any ordering Mar 16 22:53:37 vboctor In my opinion, the validation should ideally be a step that we execute before importing start. This is to avoid failing mid way during import. Specially given that we don't use transactions. Unless we are planning to use transactions. Mar 16 22:53:51 giallu so, for instance, I could ksort the array from filters and be done with this Mar 16 22:54:34 vboctor I guess ordering is a not a big deal. Mar 16 22:54:42 giallu yap. let's move on Mar 16 22:55:05 vboctor I think the best option is to have a map which defines the name mapping / ordering, but it is not a big deal. Mar 16 22:55:48 giallu so let's quicly summarize your comments: Mar 16 22:55:56 giallu 1. The export has a lot of ids. These ids may mean different things to different people working with the data. I had the same problem when defining the SOAP API interface. What I ended up doing is to use an object reference concept. Hence, rather than just using an id, I use an id and a non-localized name to refer to an object. This applies to enumerations, users, etc. Mar 16 22:56:42 thraxisp A map might also help for foreign import where fields are missing. Mar 16 22:56:58 giallu I guess I can use the same system as used in SOAP. I'll add that as needed though. is that OK? Mar 16 22:58:47 vboctor what do you mean by as needed? Mar 16 22:59:06 * nt (n=tuckern@spc1-pete6-0-0-cust391.asfd.broadband.ntl.com) has joined #mantishelp Mar 16 22:59:08 vboctor What I think we should do now is to have <reporter><id>1</id><name>administrator</name></reporter> Mar 16 22:59:34 vboctor Same for view status: <view_state><id>10</id><name>public</name></view_state> Mar 16 23:00:11 vboctor The other option is to use <view_state_id>10</view_state_id> and then have a section in the XML that defines all the possible view states. Mar 16 23:00:38 giallu it means I could assume in a first instance that we are talking about two identical installations, so states are the same Mar 16 23:01:05 vboctor But you can't assume that for users, projects, etc. Mar 16 23:01:07 giallu then we can expand to your proposal in order to support some euristic (is that the word?) Mar 16 23:01:50 vboctor how are you planning to handle users, projects, etc? Mar 16 23:01:51 giallu vboctor, no. the users are to be handled like drafted in the reqiurement page Mar 16 23:01:56 giallu i.e. Mar 16 23:02:49 * vboctor checked the wiki page. Mar 16 23:02:53 giallu define a squash-to userid during import and ask admin if everything has to be squashed to that user or just the not matching ids Mar 16 23:02:58 vboctor your sample xml didn't include user names. Mar 16 23:03:42 giallu for projects. I think I'm limiting the import to a _single_ project Mar 16 23:04:12 giallu I don't really want to guess which project an issue belongs to Mar 16 23:04:34 giallu vboctor, which names? Mar 16 23:04:58 giallu reporters? Mar 16 23:05:14 vboctor I just think having <project><id>5</id><name>Mantis</name></project> will allow us to match by name even if ids don't match. Mar 16 23:05:29 vboctor by user names, i meant reporters, handlers, etc. Mar 16 23:05:50 giallu so basically, everywhere we have an id... Mar 16 23:06:08 CIA-11 mantisbt: thraxisp * r5121 /branches/BRANCH_1_1_0/mantisbt/adm_config_report.php: Mar 16 23:06:08 CIA-11 mantisbt: partial fix for #0008976: Remote Code Execution in adm_config Mar 16 23:06:08 CIA-11 mantisbt: - hide update form for those who can't change items Mar 16 23:08:36 vboctor giallu, yes. I think it will make the format more flexible and we won't have to make breaking changes. Mar 16 23:09:24 giallu vboctor, well you know, the X in XML means we shouldn't have this problem ;) Mar 16 23:10:38 vboctor just end up with more code to maintain for no strong reason. Mar 16 23:11:02 vboctor now we need to handle <reporter_id> and <reporter> Mar 16 23:11:34 vboctor do you think it is a lot more work to export id/name combinations? Mar 16 23:11:58 vboctor or is the logic to be intelligent about such combinations on import is what is worrying you? Mar 16 23:12:36 giallu yeah. it's just more work in the import side (which is still a bit foggy in my head) Mar 16 23:12:57 giallu I think I can do that Mar 16 23:13:06 giallu just one more Q Mar 16 23:13:33 giallu do you think your 4 years old note about using DOM XML still apply? Mar 16 23:14:02 giallu if possible, I'd rather use some existing php module for the job Mar 16 23:14:27 giallu though right now I'm writing it with custom code Mar 16 23:14:35 CIA-11 mantisbt: thraxisp * r5122 /trunk/mantisbt/adm_config_report.php: Mar 16 23:14:35 CIA-11 mantisbt: partial fix for #0008980: Port: Remote Code Execution in adm_config Mar 16 23:14:35 CIA-11 mantisbt: - hide change form from unauthorized users Mar 16 23:15:17 vboctor custom code is fine. Mar 16 23:16:11 giallu yes. but I'm asking if it's fine to use some xml related functionality. The parsing is trickier Mar 16 23:17:50 vboctor I don't understand your question, probably since I don't remember the comment you are referring to. Mar 16 23:17:58 * vboctor checking... Mar 16 23:18:13 giallu ah. you asked to _not_ use DOM XML Mar 16 23:18:34 giallu becasue it was possibly not available everywhere Mar 16 23:19:42 vboctor yep, just read it now. Mar 16 23:19:48 vboctor what is the status of DOM XML and PHP 5? Mar 16 23:19:59 vboctor I remeber there was also Simple XML in PHP 5? Mar 16 23:20:36 giallu right Mar 16 23:21:02 vboctor Given that export / import is not a core feature, I am ok with using a PHP extension. Mar 16 23:21:09 giallu great Mar 16 23:21:17 vboctor Similar to twitter notifications that is dependent on curl extension. Mar 16 23:21:37 giallu I'm probably leaving the exporter as is unless I find it unmaintenable Mar 16 23:22:21 giallu but I'm definitely looking for some cooked solution about import Mar 16 23:22:43 vboctor consistency between export / import would be ideal. Mar 16 23:23:06 vboctor you also need to be careful that PHP doesn't run out of memory when exporting / importing a lot of issues. Mar 16 23:23:47 giallu ok. I'm noting this on the wiki page Mar 16 23:33:20 vboctor good