View Issue Details

IDProjectCategoryView StatusLast Update
0017577mantisbtperformancepublic2017-04-30 14:53
ReporterdregadAssigned Todregad 
PrioritynormalSeverityminorReproducibilityalways
Status assignedResolutionopen 
Product Version1.2.17 
Target Version2.5.0Fixed in Version 
Summary0017577: Improve print_user_option_list() performance
Description

On instances with large number of users, the function has very poor
performance (e.g. > 20 seconds to load 28K users on this tracker).

This is an issue e.g. when filtering issues by reporter or when using adm_config_report.php, especially when the current project is 'All Projects'.

TagsNo tags attached.

Relationships

has duplicate 0017737 closedatrol adm_config_report.php is slow with larger number of users 
has duplicate 0019942 closedatrol Reporter list is loading way too long 
related to 0017962 new Implement auto-complete for reporter field in bug_update_page 
related to 0017738 new return_dynamic_Filters.php performance with large number of users 

Activities

dregad

dregad

2014-08-07 19:49

developer   ~0041032

PR https://github.com/mantisbt/mantisbt/pull/238

The proposed fix is for master, I plan to backport it to 1.2.x if accepted.

atrol

atrol

2015-09-09 09:50

developer   ~0051427

Reminder sent to: dregad

Should we target to 1.3.x or maybe even 2.0.x?

dregad

dregad

2015-09-09 18:51

developer   ~0051429

The reason for targeting 1.2 was that our tracker was heavily affected by this issue. Now that we have upgraded it, a 1.3.x target is probably best (assuming I ever find the time to get to it :/).

The impact is still major IMO, although not blocking for release since not so many instances are likely to be affected by this.

cproensa

cproensa

2015-10-26 20:51

developer   ~0051728

for solving half of the problem: why not rewrite the user selector in filter section to fetch all users that appear in existing bugs?

This would be a query (fine tuned to some degree by filter parameters), but surely less work on the server.
The real problem here is that for "all projects", its fetching and combining a list of all users, repeating for every project.

And, for an improved resolution, change the select into an autocomplete widget (when available by js), with progressive remote search

dregad

dregad

2015-10-27 04:20

developer   ~0051732

change the select into an autocomplete widget (when available by js), with progressive remote search

This is exactly where I want to go. I'm pretty sure I have a feature branch with initial research on this somehere, but I can't find it ATM.

atrol

atrol

2015-10-27 04:39

developer   ~0051734

I'm pretty sure I have a feature branch

Maybe this one?
https://github.com/dregad/mantisbt/commits/dynamic-reporter-select

Independet from AJAX some other optimizations that did not make their way into the product but might be worth to review them again.

https://github.com/mantisbt/mantisbt/pull/238
https://github.com/mantisbt/mantisbt/pull/271

cproensa

cproensa

2015-10-27 06:48

developer   ~0051735

^^^ regarding that:
please have a look at 0020225, its somewhat related to the affected core functions

cproensa

cproensa

2015-10-29 08:04

developer   ~0051753

i have this branch with a rewrite of the affected functions
https://github.com/cproensa/mantisbt/tree/project_user_list

this shows great improvement, as testing with 20k users, 0000054:0000100 projects
still it can get faster, but i have some questions before it can be done
should i open a PR to comment there?

cproensa

cproensa

2015-10-31 21:41

developer   ~0051780

PR: https://github.com/mantisbt/mantisbt/pull/668

vboctor

vboctor

2015-12-22 03:38

manager   ~0052164

Last edited: 2016-05-09 12:03

View 2 revisions

Changing status to minor based on @dregad's 0017577:0051429 and offline discussion with @atrol since we are trying to identify blocking issues for v1.3. Though this may make it before the release, it is not blocking.

dregad

dregad

2016-05-21 04:31

developer   ~0053204

cproensa's PR https://github.com/mantisbt/mantisbt/pull/769 (see related changeset below) greatly improves this performance issue.

atrol

atrol

2016-06-14 02:17

developer   ~0053370

Independent from the AJAX approach and other optimizations, but will also enhance performance at least for some of the user lists, e.g. "Reporter" on "View Issues" page:

Wouldn't it be better to list just the users that reported an issue for the current project (distinct reporter_id from bug table)?

This will reduce the number of listed users and solve also the issue that we are not able to search for issues that have been reported by deactivated or deleted users, see 0010141.

cproensa

cproensa

2016-06-14 03:37

developer   ~0053374

distinct xxx from bug table

yes, that would be the simplest solution. For several filter fields this should be a valid approach.

able to search for issues that have been reported by deactivated or deleted users

but, irrc, then someone can complain that a reporter gets a hint of the existence of the deactivated users (i dont think this is an issue, just saying)

Now that theres some will to change things it filter api, this is a thing that should be done

atrol

atrol

2016-06-18 13:17

developer   ~0053412

someone can complain that a reporter gets a hint of the existence of the deactivated users

We could add a new option for it, something like $g_show_deactivated_users_threshold = MANAGER;
But this will not prevent a user from using "View Issues" page to list all issues and look for users which are displayed strike out.
So adding this option would just let some people think that users with access level < $g_show_deactivated_users_threshold can't find deactivated users.

dregad

dregad

2017-01-24 08:46

developer   ~0055289

Unless someone objects, I'd like to mark this one as resolved, considering cproensa's commit MantisBT master b433e7e6 - on this tracker it takes about 3-5 seconds to load the reporters' list (about 35K users), which I think is acceptable.

Other ideas for improvement should be tracked separately.

atrol

atrol

2017-01-24 15:49

developer   ~0055298

or when using adm_config_report.php, especially when the current project is 'All Projects'

This is still quite slow.
Didn't have a deeper look, might be caused by missing indices on mantis_config_table.

Related Changesets

MantisBT: master b433e7e6

2016-04-29 17:13:12

cproensa


Committer: dregad Details Diff
Don't cache users in project_get_all_user_rows()

Most of the callers of project_get_all_user_rows use the return array
directly, and do not need to fetch the user rows. There's no need to
always cache the calculated users. Especially when the user list is very
large, eg: print users option list with public access. By not caching
them, memory and time is saved (see issue 0017577)

Of all the callers of this function, those that actually use the
cached data are:
- email_notify_new_account(): Adds an explicit call to cache
- email_collect_recipients(): Was already calling to cache users

Signed-off-by: Damien Regad <dregad@mantisbt.org>
mod - core/email_api.php Diff File
mod - core/project_api.php Diff File

Issue History

Date Modified Username Field Change
2014-08-07 19:46 dregad New Issue
2014-08-07 19:46 dregad Status new => assigned
2014-08-07 19:46 dregad Assigned To => dregad
2014-08-07 19:49 dregad Note Added: 0041032
2014-08-08 05:14 fuckoff File Added: ahi_extension.xpi
2014-08-08 07:58 atrol File Deleted: ahi_extension.xpi
2014-10-09 06:02 atrol Relationship added has duplicate 0017737
2014-12-05 18:34 dregadmin Target Version 1.2.18 => 1.2.19
2014-12-14 17:49 dregad Relationship added related to 0017962
2015-01-25 18:18 dregadmin Target Version 1.2.19 => 1.2.20
2015-07-18 09:15 atrol Relationship added has duplicate 0019942
2015-09-09 09:50 atrol Note Added: 0051427
2015-09-09 18:51 dregad Target Version 1.2.20 => 1.3.0-rc.1
2015-09-09 18:51 dregad Note Added: 0051429
2015-10-26 20:51 cproensa Note Added: 0051728
2015-10-27 04:20 dregad Note Added: 0051732
2015-10-27 04:39 atrol Note Added: 0051734
2015-10-27 06:48 cproensa Note Added: 0051735
2015-10-29 08:04 cproensa Note Added: 0051753
2015-10-31 21:41 cproensa Note Added: 0051780
2015-12-06 02:55 vboctor Target Version 1.3.0-rc.1 => 1.3.0-rc.2
2015-12-22 03:38 vboctor Severity major => minor
2015-12-22 03:38 vboctor Note Added: 0052164
2016-05-09 12:03 dregad Note Edited: 0052164 View Revisions
2016-05-15 09:55 dregad Changeset attached => MantisBT master b433e7e6
2016-05-21 04:31 dregad Note Added: 0053204
2016-05-31 10:33 cproensa Relationship added related to 0017738
2016-06-12 02:37 atrol Target Version 1.3.0-rc.2 => 1.3.0
2016-06-14 02:17 atrol Note Added: 0053370
2016-06-14 03:37 cproensa Note Added: 0053374
2016-06-18 13:17 atrol Note Added: 0053412
2016-07-10 07:57 atroladmin Target Version 1.3.0 => 1.3.1
2016-08-28 10:37 atrol Target Version 1.3.1 => 1.3.2
2016-10-02 19:36 atrol Target Version 1.3.2 => 1.3.3
2016-10-30 23:23 vboctor Target Version 1.3.3 => 1.3.4
2016-11-27 08:22 dregad Target Version 1.3.4 => 1.3.5
2016-12-30 16:24 atrol Target Version 1.3.5 => 1.3.6
2017-01-24 08:46 dregad Note Added: 0055289
2017-01-24 15:49 atrol Note Added: 0055298
2017-02-01 22:47 vboctor Target Version 1.3.6 => 1.3.7
2017-02-26 21:19 vboctor Target Version 1.3.7 => 2.3.0
2017-04-01 00:20 vboctor Target Version 2.3.0 => 2.4.0
2017-04-30 14:53 vboctoradmin Target Version 2.4.0 => 2.5.0