View Issue Details

IDProjectCategoryView StatusLast Update
0031213mantisbtapi restpublic2022-11-16 14:04
Reportersboucard Assigned Toatrol  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionno change required 
Product Version2.22.2 
Summary0031213: Inconsistent paging with REST issues end point
Description

REST "issues" end point gives the ability to list issues of a given project or for a given filter. This end point supports paging, whose size is controlled by the "page_size" query parameter. The client performs a GET request on this end point with a "page" query parameter that starts with 1 and is incremented for next GET request, until the response no longer contains any issue.
We experience an inconsistent list of issues with this paging mechanism:

  • Some issues appear in one page but also in next page
  • Some issues do not appear in any page, though they are part of the project/filter
    It seems to be really a paging problem. Executing the same sequence of GET requests but with different page sizes reveals a different subset of issues being duplicated/non-present.
    It is worth mentioning the defect is constant over executions: for a given page size, it is always the exact same subset of issues that are duplicated/non-present.
Steps To Reproduce

Use a REST client to iterate on GET responses and compare the total list of issue IDs using different page sizes. Different page sizes retrieve a different list of issues.
We have elaborated a Java snippet on our side demonstrating the defect. I'm willing to share if it helps your investigation.

TagsNo tags attached.

Activities

atrol

atrol

2022-10-28 07:10

developer   ~0067108

Are you able to reproduce using latest stable version? (2.25.5. at the moment)
I didn't have a deeper look, but there were some REST and filter related changes since 2.22.2.

dregad

dregad

2022-10-31 10:47

developer   ~0067114

We have elaborated a Java snippet on our side demonstrating the defect. I'm willing to share if it helps your investigation.

That would be nice, please do share. And if you could make the script target this tracker so we have a reproducible test case, it would be even better.

sboucard

sboucard

2022-10-31 11:19

reporter   ~0067115

And if you could make the script target this tracker...
Oh, is REST really active on https://www.mantisbt.org/? I get HTTP 404 trying to use that.

I'm trying to get my IT to upgrade Mantis to 2.25.5, that may take a few days.

Attached the IssuePagingTest.java file. This (rough and quickly designed) Java program requires Java 11 and Jakarta JSON library (https://repo1.maven.org/maven2/org/glassfish/jakarta.json/2.0.1/jakarta.json-2.0.1.jar) in the classpath.
Its takes as argument:

  • The base URL of the Mantis server (e.g. https://mymantis.myserver.cloud/)
  • The token to use REST APIS
  • The identifier of the Mantis project whose issues are desired (e.g. 48)
    The program uses different page sizes and iterates on obtained pages.
    It reports duplicated issues as well as different issue lists obtained with different page sizes.
    Hopefully it helps you to reproduce and root cause the problem. Or to spot what I did wrong on my end ;-)
IssuePagingTest.java (5,508 bytes)   
package test.issue_paging;

import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.stream.Collectors;

import jakarta.json.Json;
import jakarta.json.JsonObject;

public class IssuePagingTest implements Callable<Void> {

    public static void main(String[] args) throws Exception {
        String mantisUrl = args[0];
        String mantisToken = args[1];
        String projectId = args[2];
        new IssuePagingTest(mantisUrl, mantisToken, projectId).call();
    }

    private String mantisUrl;
    private String mantisToken;
    private String projectId;
    private HttpClient httpClient;

    private IssuePagingTest(String mantisUrl, String mantisToken, String projectId) {
        this.mantisUrl = mantisUrl;
        this.mantisToken = mantisToken;
        this.projectId = projectId;
        this.httpClient = HttpClient.newBuilder().version(HttpClient.Version.HTTP_1_1).connectTimeout(Duration.ofSeconds(10)).build();
    }
    
    @Override
    public Void call() throws Exception {
        System.out.println("Mantis URL: " + mantisUrl);
        System.out.println("Project: " + projectId);
        
        Set<Integer> total = new HashSet<>();
        Set<Integer> issues35 = getProjectIssues(35);
        Set<Integer> issues50 = getProjectIssues(50);
        Set<Integer> issues80 = getProjectIssues(80);
        
        System.out.println("---------------------------------");
        System.out.println("Compare issues with page sizes 35 and 50:");
        compare(issues35, issues50);
        System.out.println("Compare issues with page sizes 50 and 80:");
        compare(issues50, issues80);
        System.out.println("Compare issues with page sizes 35 and 80:");
        compare(issues50, issues80);
        
        total.addAll(issues35);
        int totalSize = total.size();
        total.addAll(issues50);
        if (totalSize < total.size()) {
            System.out.println((total.size() - totalSize) + " issues added with paging 50");
        }
        totalSize = total.size();
        total.addAll(issues80);
        if (totalSize < total.size()) {
            System.out.println((total.size() - totalSize) + " issues added with paging 80");
        }
        return null;
    }
    
    private void compare(Set<Integer> issues1, Set<Integer> issues2) {
        Set<Integer> removed = new HashSet<>(issues1);
        removed.removeAll(issues2);
        if (!removed.isEmpty()) {
            System.out.println(removed.size() + " removed issues: " + removed);
        }
        
        Set<Integer> added = new HashSet<>(issues2);
        added.removeAll(issues1);
        if (!added.isEmpty()) {
            System.out.println(added.size() + " added issues: " + added);
        }
    }

    private HttpResponse<InputStream> get(String endpoint) throws IOException {

        HttpRequest request = HttpRequest.newBuilder().GET().uri(URI.create(mantisUrl + "api/rest" + endpoint)).header("Authorization", mantisToken)
                .build();

        try {
            HttpResponse<InputStream> response = httpClient.send(request, HttpResponse.BodyHandlers.ofInputStream());

            // success?
            if (response.statusCode() < 300) {
                return response;
            }

            // fail with an exception if the Mantis server responded an HTTP error
            else {
                throw new RuntimeException("HTTP " + response.statusCode());
            }
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    private Set<Integer> getProjectIssues(int pageSize) throws IOException {
        Set<Integer> totalIssues = new HashSet<>();
        int pageCount = 0;
        List<Integer> ids;

        System.out.println("---------------------------------");
        System.out.println("Querying issues with page size " + pageSize);
        long start = System.currentTimeMillis();
        do {
            // execute the GET
            pageCount++;
            System.out.println("Querying issue page " + (pageCount) + "...");
            String queryParams = String.format("project_id=%s&page_size=%d&page=%d", projectId, pageSize, pageCount);
            HttpResponse<InputStream> response = get(
                    "/issues/?" + queryParams);

            // process the JSON response
            try (InputStream is = response.body()) {
                JsonObject root = Json.createReader(is).readObject();
                ids = new ArrayList<>(
                        root.getJsonArray("issues").stream().map(issue -> ((JsonObject) issue).getInt("id")).collect(Collectors.toList()));

                for (Integer id : ids) {
                    if (!totalIssues.add(id)) {
                        System.err.println("Duplicate issue in response: " + id);
                    }
                }
            }
        } while (ids.size() > 0);

        System.out.println("Issues: " + totalIssues.size());
        System.out.println("Done in " + (System.currentTimeMillis() - start) + "ms");
        return totalIssues;
    }
}
IssuePagingTest.java (5,508 bytes)   
dregad

dregad

2022-10-31 11:58

developer   ~0067118

Oh, is REST really active on https://www.mantisbt.org/? I get HTTP 404 trying to use that.

https://mantisbt.org/bugs/api/rest/issues/31213 works for me...

Attached the IssuePagingTest.java file

Thanks. I don't know much Java, but hopefully enough to understand what you're doing ;-)

No promises on when I can take a close look at it though...

sboucard

sboucard

2022-10-31 12:31

reporter   ~0067120

You're right, I can indeed use REST on https://mantisbt.org/bugs/
And that is interesting as my snippet does not exhibit any problem with that tracker.
So it is indeed possible the paging issue was somehow fixed in 2.25.5 (or in that range between the version I use and this one).
My next task is to perform the Mantis upgrade on my side and verify the problem is gone.
I keep you posted.

sboucard

sboucard

2022-11-04 05:20

reporter   ~0067124

I confirm I no longer experience the paging problem using latest and greatest of Mantis.
Thank you for the hints.
You shall mark this ticket as resolved.