User Tools

Site Tools


mantisbt:database_optimization

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
mantisbt:database_optimization [2006/10/30 06:09]
RogerHill
mantisbt:database_optimization [2008/10/29 04:25] (current)
Line 1: Line 1:
 +====== Database Optimization ======
 +
 This page describes some database optimisations for SQL Server 2000. These are based on Mantis 1.0.2 This page describes some database optimisations for SQL Server 2000. These are based on Mantis 1.0.2
  
Line 4: Line 6:
  
 The project I was working on used a Mantis installation based on Windows 2003 Server/IIS/PHP and SQL Server. The project I was working on used a Mantis installation based on Windows 2003 Server/IIS/PHP and SQL Server.
-The filters were starting to become unusably slow, particularly the view_all_bug_page. Pressing the "Rest Filter" button could produce a delay of 40-50 seconds before a page appeared. This problem was getting worse as we added more and more issues.+The filters were starting to become unusably slow, particularly the view_all_bug_page. Pressing the "Reset Filter" button could produce a delay of 40-50 seconds before a page appeared. This problem was getting worse as we added more and more issues.
  
 ===== The Investigation ===== ===== The Investigation =====
  
-The investigation of the problem centered on the filter code in code/filter_api.php. The function filter_get_bug_rows() is the core of the bug filtering system.+The investigation of the problem centered on the filter code in core/filter_api.php. The function filter_get_bug_rows() is the core of the bug filtering system.
  
 The existing filter_get_bug_rows() function is 813 lines of code - arguably a candidate for some "refactoring with extreme prejudice" however that is beyond the scope of what I was trying to do here, which was to make MANTIS work acceptably under SQL server with more than 1000 open issues. The existing filter_get_bug_rows() function is 813 lines of code - arguably a candidate for some "refactoring with extreme prejudice" however that is beyond the scope of what I was trying to do here, which was to make MANTIS work acceptably under SQL server with more than 1000 open issues.
Line 14: Line 16:
 What is interesting is the area after line 724 in the original file. What happens here is that a list of relevant issues is created in the variable $t_id_array. This is then converted into a where clause at line 756. This produces a SQL statement which will be used to query mantis_bug_table of the form: - What is interesting is the area after line 724 in the original file. What happens here is that a list of relevant issues is created in the variable $t_id_array. This is then converted into a where clause at line 756. This produces a SQL statement which will be used to query mantis_bug_table of the form: -
  
-''+<code sql>
 select ... from mantis_bug_table join ......  select ... from mantis_bug_table join ...... 
- where  +  where  
- mantis_bug_table.id in ( 1, 10, 11, ...... long list of bug ids ... ) +  mantis_bug_table.id in ( 1, 10, 11, ...... long list of bug ids ... ) 
-''+</code> 
 + 
  
 The key observation that I made was that SQL Server **hates** long in (...)  lists.  The key observation that I made was that SQL Server **hates** long in (...)  lists. 
Line 25: Line 29:
 ===== The Solution ===== ===== The Solution =====
  
-Given that the problem was caused by the excessively long in() clause, my solution was to convert the in(...) clause to a temporary table, so that the SQL show above becomes...+Given that the problem was caused by the excessively long in() clause, my solution was to convert the in(...) clause to a temporary table, so that the SQL shown above becomes...
  
-''+<code sql>
 select ... from mantis_bug_table join ......  select ... from mantis_bug_table join ...... 
  #filtered_ids fid on fid.id = mantis_bug_table.id  #filtered_ids fid on fid.id = mantis_bug_table.id
-''+</code>
  
-Where the #filtered_ids table contains the list of bug_ids that were previously in the in(...) clause.+Where the #filtered_ids table contains the list of bug_ids that were previously in the in(...) clause. This temporary table id also indexed and clustered on the id field so that we can squeeze the last drop of performance out of the table index.
  
 The full text of my solution is shown below: - The full text of my solution is shown below: -
  
-Firstly replace the code after line 724 with the following : -+Firstly replace the code from line 724 to 764 with the following : -
  
-''+<code php>
  # Possibly do two passes. First time, grab the IDs of issues that match the filters. Second time, grab the IDs of issues that  # Possibly do two passes. First time, grab the IDs of issues that match the filters. Second time, grab the IDs of issues that
  # have bugnotes that match the text search if necessary.  # have bugnotes that match the text search if necessary.
Line 86: Line 90:
  # Get the total number of bugs that meet the criteria.  # Get the total number of bugs that meet the criteria.
  $bug_count = $row_count; #count( $t_id_array );  $bug_count = $row_count; #count( $t_id_array );
-''+</code>
  
-Then at line 853 change the SELECT to :- +Then at line 853 change the SELECT statement to :- 
  
  
-''+<code php>
  $t_order = " ORDER BY " . implode( ', ', $t_order_array );  $t_order = " ORDER BY " . implode( ', ', $t_order_array );
  $t_select = implode( ', ', array_unique( $t_select_clauses ) );  $t_select = implode( ', ', array_unique( $t_select_clauses ) );
Line 105: Line 109:
  # Figure out the offset into the db query  # Figure out the offset into the db query
  #  #
-''+</code> 
 + 
 + 
 +===== Alternative solution by RZG ===== 
 + 
 +NOTE BY RZG: 
 +My IMHO simpler solution was to use subqueries. With just 2 small changes, we were able to fix the issues described on my company Mantis, which has well over 10000 issues on it. See [[http://www.mantisbt.org/bugs/view.php?id=7516|Issue 7516]].
  
 ===== How much difference does it make? ===== ===== How much difference does it make? =====
Line 111: Line 121:
 I will admit that I haven't made any exact measurements of the difference that this optimization makes with out current load of over  6400 open issues.  I will admit that I haven't made any exact measurements of the difference that this optimization makes with out current load of over  6400 open issues. 
  
-What I do know is that when we had about 1000 open issues, the view_all_bug_page was regularly taking over 40s to appear. (Sometimes the SQL caching would work, and it would appear straight away, other times, when a newly added bug changed the text of that in(...) clause the SQL would change, and everything would slow down again...)+What I do know is that when we had about 1000 open issues, the view_all_bug_page was regularly taking over 40s to appear. (Sometimes the SQL caching would work, and it would appear straight away, other times, when a newly added bug changed the text of that in(...) clause the SQL would change, and everything would slow down again...) To add to that the time taken to display the view_all_bug_page seemed to be increasing in a non-linear fashion. The result for the project would have been catastrophic as the number of issues continued to rise.
  
 In addition to the above SQL optimisation, there were some significant benefits to be gained from adding cacheing for custom field values that we have displayed in the view_all_bugs_page. In addition to the above SQL optimisation, there were some significant benefits to be gained from adding cacheing for custom field values that we have displayed in the view_all_bugs_page.
Line 117: Line 127:
 We now have over 6400 open issues. The view_all_bugs_page never takes more than 5 seconds to appear. Pages other than the  first  page such as accessed by clicking on the "Next" link can be accessed in under 3 seconds. We now have over 6400 open issues. The view_all_bugs_page never takes more than 5 seconds to appear. Pages other than the  first  page such as accessed by clicking on the "Next" link can be accessed in under 3 seconds.
  
 +===== Future developments =====
  
 +This modification has a number of limitations, particularly in that it is very much focused on SQL Server syntax. Ideally we would have an architecture in place in filter_api.php that allowed different database backends to use their own optimised search methods. This would probably require the implementation of architectural changes to MANTIS to allow for the use of variations in the query strategy with different SQL dialects. 
  
- +For SQL server, I think we would want to look at using a stored procedure to return the required dataset for a particular page in view_all_bugs_page.
- +
- +
- +
- +
- +
  
mantisbt/database_optimization.1162206587.txt.gz · Last modified: 2008/10/29 04:31 (external edit)