View Issue Details

IDProjectCategoryView StatusLast Update
0019267mantisbtupgradepublic2016-08-03 10:37
Reportervboctor Assigned Tovboctor  
PrioritynormalSeverityblockReproducibilityalways
Status closedResolutionwon't fix 
Product Version1.3.0-beta.1 
Summary0019267: Upgrade converts complex db configs to NULLs
Description

If the mantisbt 1.2.x database has complex config options, such options are converted to NULLs as part of the upgrade causing the application to break in random places and the configuration to be lost.

Tagsmantishub

Relationships

related to 0021312 confirmed Error Schema step 193 - Upgrading 1.2.19 to 1.3.0 
related to 0021376 closeddregad Error in upgrade process 1.2.17 --> 1.3.0 

Activities

vboctor

vboctor

2015-01-26 08:35

manager   ~0048692

Deleting the null configs makes MantisBT work again, but the configuration is lost. It is also hard for users to discover the issue.

vboctor

vboctor

2015-01-27 18:26

manager   ~0048707

I looked into this further and here are the observations:

  • The value in the database is not actually null, but the old php serialize() format.
  • The reason the database ends in this state, is that there is at least one bad complex value that unserialize() can't decode.
  • Once we hit one bad complex value, we don't even try the rest and assume there is a code bug. Fact is there are cases where for some reason the serialized value is invalid.
  • I've fixed this by unserialize() + json_encode() for all entries. For the ones that fail, I just deleted them.

Having such invalid configs generates all sort of random errors in the apps, since other APIs gets a null value for config thresholds or workflows and passes these to APIs that don't expect nulls.

My recommendation here is to follow the model:

  • attempt to upgrade all complex fields.
  • when failed, show an error and continue.

If we want beta1 users to benefit from this, then we can add this as another step that detects complex fields with value starting with "a:" (for array) and upgrade them. If unserialize() fails, then delete.

To reproduce this create a 1.2.x instance with some of these complex configs, corrupt one of them and then upgrade to 1.3.

dregad

dregad

2015-01-28 04:34

developer   ~0048720

I'm curious as to how you got there in the first place, considering that the install_check_config_serialization() upgrade function should fail with an error if it detects an invalid config (leaving you with an incomplete upgrade).

Continuing after an error is incorrect, and IMO having the installer deleting an offending (corrupted) value may be a problem, so I think it's best to leave fixing such issues to manual correction by the admin.

What we should do is make it clear to the admin what why the error is occuring, and give them options (what they can do to fix it).

I'm not sure how this could be done in the installer, as by nature the options are fairly limited in terms of what we can display back to the admin in case of error. Maybe an admin check ?

vboctoradmin

vboctoradmin

2015-01-28 11:42

administrator   ~0048721

The upgrade was done using upgrade_unattended script. I'm not sure whether the script stopped on first error or just ignore that this step is failing and continued other steps.

If we assume that the customer has a backup and that configs is not really the end of the world, then we may delete the bad items and provide the user with a list of what we have deleted after the fact. That wouldn't help with the upgrade_unattended script though.

The other issue that I'm not sure about is why unserialize() is sometimes not able to deserialize its own data. And why such configs didn't cause an issue before the upgrade. i.e. the blob was invalid. Then even with old 1.2.x code, the app should have got some bad value (e.g. false/null, etc) that would have impacted the logic of the app.

dregad

dregad

2015-01-28 19:22

developer   ~0048724

If we assume that the customer has a backup

Ahem. Well, you know... I thought I had one, but...
;-)

Joke aside, do you have some specific example(s) of corrupted entries ? I'd like to have a look at the upgrade_unattended script with a real test case (I never actually used it myself). It'd also be interesting to see what happens in 1.2.x with such corrupted data.

vboctor

vboctor

2015-01-29 10:53

manager   ~0048732

Last edited: 2015-03-14 14:09

View 2 revisions

Here is a sample corrupt filter:

Corruption caused by special characters next to "+educa" - remove the "v8#" prefix and then test the result with unserialize().

<pre>
v8#a:44:{s:13:"show_category";a:1:{i:0;s:1:"0";}s:13:"show_severity";a:1:{i:0;i:0;}s:11:"show_status";a:1:{i:0;i:0;}s:17:"highlight_changed";i:6;s:11:"reporter_id";a:1:{i:0;i:0;}s:10:"handler_id";a:1:{i:0;i:0;}s:10:"project_id";i:0;s:15:"show_resolution";a:1:{i:0;i:0;}s:10:"show_build";a:1:{i:0;s:1:"0";}s:12:"show_version";a:1:{i:0;s:1:"0";}s:11:"hide_status";a:1:{i:0;i:90;}s:12:"user_monitor";a:1:{i:0;i:0;}s:4:"sort";s:12:"last_updated";s:3:"dir";s:4:"DESC";s:8:"per_page";i:50;s:10:"match_type";i:0;s:8:"_version";s:2:"v8";s:10:"_view_type";s:8:"advanced";s:13:"sticky_issues";i:1;s:8:"platform";i:0;s:2:"os";i:0;s:8:"os_build";i:0;s:11:"start_month";i:0;s:9:"start_day";i:0;s:10:"start_year";i:0;s:9:"end_month";i:0;s:7:"end_day";i:0;s:8:"end_year";i:0;s:6:"search";s:0:"";s:16:"and_not_assigned";b:0;s:17:"do_filter_by_date";b:0;s:10:"view_state";i:0;s:17:"relationship_type";i:-1;s:16:"relationship_bug";i:0;s:14:"target_version";a:1:{i:0;s:1:"0";}s:10:"tag_string";s:23:"+multimidia,+educa��o";s:10:"tag_select";s:0:"";s:12:"note_user_id";a:1:{i:0;i:0;}s:13:"show_priority";a:1:{i:0;i:0;}s:16:"fixed_in_version";a:1:{i:0;s:1:"0";}s:12:"show_profile";a:1:{i:0;i:0;}s:13:"custom_fields";a:4:{i:4;a:1:{i:0;s:1:"0";}i:1;a:1:{i:0;s:1:"0";}i:3;a:1:{i:0;s:1:"0";}i:2;a:1:{i:0;s:1:"0";}}s:10:"profile_id";i:0;s:16:"_source_query_id";s:0:"";}
</pre>

dregad

dregad

2015-03-14 15:25

developer   ~0049231

Just tested this a bit more in depth:

  1. setup a new 1.2.x DB
  2. set a complex config, e.g. by customizing workflow
  3. manually edit the config (via phpMyAdmin) to make it corrupt

At this stage, the adm_config_report.php page will show the config as corrupted, and depending on error settings, a PHP notice will be displayed on some screens:
<pre>
All Users Test project status_enum_workflow complex The configuration in the database is corrupted. administrator
</pre>

SYSTEM NOTICE: 'unserialize(): Error at offset 312 of 548 bytes' in '/home/dregad/dev/mantisbt/core/config_api.php' line 161

  1. run the upgrade (interactively) - fails with error

<pre>Schema step 193: UpdateFunction ( check_config_serialization )
SYSTEM NOTICE: 'unserialize(): Error at offset 312 of 548 bytes' in '/home/dregad/dev/mantisbt/core/install_helper_functions_api.php' line 644
BAD
check_config_serialization
Array
</pre>

  1. run upgrade_unattended.php - fails with error (on command-line, script exits with error code 1)

<pre>
Post 1.0 schema changes
Connecting to database... OK
Schema UpdateFunction ( check_config_serialization ) - ERROR: check_config_serialization
1 schema upgrades executed.
Failed.
</pre>

Based on the above, as far as I can tell the system works as designed, and no change is required.

Let me know your thoughts.

Issue History

Date Modified Username Field Change
2015-01-26 08:34 vboctor New Issue
2015-01-26 08:34 vboctor Tag Attached: mantishub
2015-01-26 08:35 vboctor Status new => confirmed
2015-01-26 08:35 vboctor Note Added: 0048692
2015-01-27 18:26 vboctor Note Added: 0048707
2015-01-28 04:34 dregad Note Added: 0048720
2015-01-28 11:42 vboctoradmin Note Added: 0048721
2015-01-28 19:22 dregad Note Added: 0048724
2015-01-29 10:53 vboctor Note Added: 0048732
2015-03-14 14:09 dregad Note Edited: 0048732 View Revisions
2015-03-14 15:25 dregad Note Added: 0049231
2015-03-15 19:59 dregad Target Version 1.3.0-beta.2 => 1.3.0-beta.3
2015-04-18 18:32 vboctor Assigned To => vboctor
2015-04-18 18:32 vboctor Status confirmed => resolved
2015-04-18 18:32 vboctor Resolution open => won't fix
2015-04-19 06:34 atrol Target Version 1.3.0-beta.3 =>
2015-04-29 13:19 atrol Status resolved => closed
2016-07-18 13:33 atrol Relationship added related to 0021312
2016-08-03 10:37 vboctor Relationship added related to 0021376