Git integration using source-integration plugin fails

General discussion about MantisBT Plugins

Moderators: Developer, Contributor

Post Reply
mntse
Posts: 7
Joined: Jan 02, 2019 3:11 pm

Git integration using source-integration plugin fails

Post by mntse » Jan 02, 2019 3:58 pm

Hi.
Before posting this I tried my best to search here and there the answers for my questions. I failed, therefore decided to ask for assistance here from more advanced Mantis users. Hope this is a right place.
I would like to integrate local Git with my local Mantis installation. This is few months old fresh install of 2.16.0 Mantis on Ubuntu 18.04 Server with Postgresql database and delivered via Apache https. source-integration plugin is at the 2.1.5 version. This is intranet only installation therefore cgit v 1.1 was installed and it is serving a test git repo webpage: repo name at index level, master branch at summary level, clone url, proper commits at log level, proper data (author, committer, commit, tree) at commit level. All good using cgit in the browser.
cgit repo in Mantis is configured:
Name: cgit test
Type: Cgit
URL: https://192.168.0.1/cgit/mantisrepotest/
cgit_root: https://192.168.0.1/cgit/
cgit_project: mantisrepotest
master_branch: master

Clone url is https://192.168.0.1/cgit/mantisrepotest
I can clone using git from this address without problems.

Unfortunately SourceCgit plugin doesn't work when using Import Everything/Import Latest Data button.
I am receiving the following error:
APPLICATION ERROR #401
Database query failed. Error received from database was #-1: ERROR null value in column "revision" violates not-null constraint.
DETAIL: Failing row contains (14, 14, null, master, 0, 1970-01-01 01:00:00...... null , null, null, 0). for the query: INSERT INTO mantis_plugin_Source_changeset_table (repo_id, revision, parent, branch, user_id, timestamp, author, message, info, ported, author_email, committer, committer_email, committer_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14).

I have found this script SourceCgit/SourceCgitTest.php but trying to run it from its directory with php SourceCgitTest.php gives me PHP Fatal error: Class 'MantisSourceGitBasePlugin' not found in /var/mantis/plugins/SourceCgit/SourceCgit.php on line 17.
I can't troubleshoot this and check if proper data is screenscrapped from my cgit service (before run I updated this file with my real data working commit string).
Hope someone can help me with cgit or plain Git configuration.
P.S.
Other Mantis plugins are working correctly. I managed to import some issues using them. There are no errors at admin check pages. Installation is working fine. Active Directory logins are working. All good apart Git....

Starbuck
Posts: 180
Joined: Feb 13, 2006 9:53 pm
Location: USA
Contact:

Re: Git integration using source-integration plugin fails

Post by Starbuck » Jan 02, 2019 10:31 pm

I need to do this soon too but with a different Git repo, so I'm interested in what you're reporting. I've read your notes carefully and understand what you're saying. I'm not familiar with the source-integration plugin, and have never used Cgit, so I can't speak with any experience on this.

I do note however, that the error seems weird.

INSERT INTO mantis_plugin_Source_changeset_table (repo_id, revision, parent, branch, "
^^ revision is second parameter

" Failing row contains (14, 14, null, master, "
^^ you're saying the data starts like this, revision is 14? Other parameters look right.

" ERROR null value in column "revision" violates not-null constraint. "

So, 14 isn't null but I'd question if your repo ID is 14 AND your commit/revision ID is also 14. Is that just a remarkable coincidence?

There was a similar message reported in the following inquiries but these problems were never resolved:
https://github.com/mantisbt-plugins/sou ... issues/149
https://github.com/mantisbt-plugins/sou ... issues/165
https://sourceforge.net/p/mantisbt/mail ... /29547560/
https://www.bountysource.com/issues/431 ... -revisions

My guess is that you might have tripped on a very old bug AND you might have invalid data.

The Cgit plugin seems to work by parsing the HTML the comes back from the pages you've described. The code for parsing the revision is here. I'm thinking Cgit was changed to invalidate that regex, or the regex was probably never right in the first place. Similar parsing is done here. I think the error you're seeing comes after entering the request here with an empty revision.

I believe the INSERT statement you referred to was generated here. It would help to really verify that the value of the revision is null at this point and then follow it backwards.

Finally, here is an "oldie but goodie" blog by John Reese about Git integration with Mantis.

Are we any closer to solving this Very old problem? Good luck!
If you want Mantis to work differently, use or create a plugin. Visit the Plugins forums.
Ask developers to create a plugin that you need - and motivate them to help you!

Starbuck
Posts: 180
Joined: Feb 13, 2006 9:53 pm
Location: USA
Contact:

Re: Git integration using source-integration plugin fails

Post by Starbuck » Jan 02, 2019 10:54 pm

Looking at John's other notes on this topic:
If using Gitweb, versions newer than 1.6 may not work correctly, as the Gitweb plugin is scraping HTML from the viewer, and changes to the HTML structure will break the Gitweb plugin.
-- https://jreese.sh/blog/source-integration-triage

That confirms one of my suspicions. While this page says "Gitweb is the default web interface that comes with Git and is the basis for other Git scripts like cgit and gitosis" - that might be inaccurate, but it seems to be agreed that Cgit "understands" Gitweb. Well, if Cgit can parse Gitweb and Gitweb handling from this plugin is recognized as probably not working, then I think we can conclude that the Cgit plugin may not work for the same reasons.

Another tidbit of info on SourceIntegration:
https://mantisbt.org/blog/archives/mantisbt/83
And John's old blog has some nice articles on Mantis and Source Integration in particular.
If you want Mantis to work differently, use or create a plugin. Visit the Plugins forums.
Ask developers to create a plugin that you need - and motivate them to help you!

mntse
Posts: 7
Joined: Jan 02, 2019 3:11 pm

Re: Git integration using source-integration plugin fails

Post by mntse » Jan 03, 2019 11:09 am

First of all thank you Starbuck, as you invested a lot amount of your time answering me. Thank you very much once again.
" Failing row contains (14, 14, null, master, "
^^ you're saying the data starts like this, revision is 14? Other parameters look right.
" ERROR null value in column "revision" violates not-null constraint. "
So, 14 isn't null but I'd question if your repo ID is 14 AND your commit/revision ID is also 14. Is that just a remarkable coincidence?
I noticed that every time I try to import data, the first number is getting +1. I have made two runs for purposes of this report and first one begins with Failing row contains (13, 14, null, master etc. and second with Failing row contains (14, 14, null, master etc. I tried a lot of times, so it is possible that I have reached 13th and 14th instance of some data. All other error data are the same.

My test git repo has only three simple commits adding and manipulating one textfile. Simple stuff.

I will try to gather some more information, like checking the plugin table in Mantis database (as it seems that incremental number is saved somewhere) and add $g_show_detailed_errors = ON;. Also will try to inspect cgit html output and compare it with the regexes you mentioned.
Unfortunately I can't read php code, so if I could run php SourceCgitTest.php I coud see what is read from the cgit page.

I fought a lot of battles with open source code here and there, so will not abandon this just yet...

Starbuck
Posts: 180
Joined: Feb 13, 2006 9:53 pm
Location: USA
Contact:

Re: Git integration using source-integration plugin fails

Post by Starbuck » Jan 03, 2019 1:54 pm

For that query, the parameters are ( Repo_ID, Revision, ...

If it's scraping the page of a single repository, the Repo_ID should be a single unique ID for that repository. It should not be changing.
Similarly, the Revision in this context is the same as a Commit, and I don't know about Cgit but Git uses a long hex string as a GUID - I don't think an integer value is correct.

A screenshot of the listing of commits would be helpful but I think I know what's happening. The web page might look like this:

Code: Select all

Index   UserID   Revision     Branch     ....
1       14       0ce4d51      master
2       14       e7d90a1      master
So it's scraping those values, just not the right ones.
Can you post a screenshot of the page that we think it's using for source data?
If you want Mantis to work differently, use or create a plugin. Visit the Plugins forums.
Ask developers to create a plugin that you need - and motivate them to help you!

mntse
Posts: 7
Joined: Jan 02, 2019 3:11 pm

Re: Git integration using source-integration plugin fails

Post by mntse » Jan 03, 2019 4:05 pm

Thank you for your reply. My Cgit site is very plain and simple, just very similar to https://git.zx2c4.com/cgit/ (I don't have about and stats subpages, Tag section at summary page):

Code: Select all

[summary page]

Branch		Commit message	Author	Age
master		Ver 2.0		root	3 months

Age		Commit message	Author
2018-10-17	Ver 2.0		root	
2018-10-17	Ver 1.0		root
2018-10-17	Repo init	root

[log page]
Age		Commit message (Expand)		Author
2018-10-17	Ver 2.0 HEAD master		root	
2018-10-17	Ver 1.0				root
2018-10-17	Repo init			root

[commit page]
author		root <email> 2018-10-17 [time]
committer	root <email> 2018-10-17 [time]
commit		70d...string... (patch)
tree		65...string...

Repo init

Diffstat
[stuff]
That is it. Hope to find a place for some images next time...
As you can see there is no 14 or 1970-01-01 01:00:00 at any screen. I don't have an idea where those values come from and why are displayed in the error message. Those might be some Mantis settings data.
I wonder what design the Cgit plugin expects. I will try to catch the contents of variables of SourceCgit.php after each function somehow and check the html code.

Starbuck
Posts: 180
Joined: Feb 13, 2006 9:53 pm
Location: USA
Contact:

Re: Git integration using source-integration plugin fails

Post by Starbuck » Jan 03, 2019 5:41 pm

RE the date: 1970-01-01 01:00:00
That's the integer value for 0, displayed in the date format, with a 1-hour offset.
So whatever column it's using for date, it's getting 0 or maybe text which it is then converting to 0. While I'm not 100% certain, this seems to be just another indication that the columns are wrong.

Let's not try to interpret what it does with bad data. Let's try to find the good data.

It would help a lot if you can verify without question the HTML that it retrieves before it begins parsing. Then we can see if the required data is still present, and then come up with some regex to parse it out. It's possible that after some Cgit revisions that the data we need isn't there anymore, it might be in some other page. So this could get complex.

Please do what you can to find and export/log the HTML that it's using for source data. If you can get the URL that it's calling then you can just regen the page.

I'm no expert here, but I will try to help. I invite you to PM me with the URL or a screenshot of the page. I'll see if I can come up with some new regex. OR better, I'll setup a test environment and run it with your configuration, just the detail you must provide which is required to access your repo. Then I'll walk through the code to see exactly what it's doing. I can't offer to setup a full Cgit installation, sorry.
If you want Mantis to work differently, use or create a plugin. Visit the Plugins forums.
Ask developers to create a plugin that you need - and motivate them to help you!

mntse
Posts: 7
Joined: Jan 02, 2019 3:11 pm

Re: Git integration using source-integration plugin fails

Post by mntse » Jan 26, 2019 1:06 pm

Minor update - I still don't have it working yet (no time), but hope my findings will get me there soon.
First of all there is a lack of documentation how to get it working and on its dependencies. I figured out that SourceCgit, SourceBitBucket, SourceGitphp, SourceHgWeb, SourceGitweb modules are using url_get function from core/url_api.php which is a call to PHP curl extension. Unfortunately Mantis Admin Guide doesn't mention about this dependency:
http://www.mantisbt.org/docs/master/en- ... s.software

Optional extensions

Curl - required for the Twitter integration feature
Even worse the plugin's documentation doesn't write about it:
Requirements

The Source Integration framework requires MantisBT version 2.0.1 or higher.

Additionally, the SourceBitBucket and SourceGithub plugins require the PHP Curl extension, or the ability to execute system calls (shell_exec).
After installing it I have curl errors in my Apache error.log after hitting Import Everything/Import Latest Data button. curl doesn't like my self-signed certificate in its default out of the box config. I will fix it soon and hope that the plugin will start to work.

mntse
Posts: 7
Joined: Jan 02, 2019 3:11 pm

Re: Git integration using source-integration plugin fails

Post by mntse » Jan 28, 2019 10:43 am

After adding my self-signed cert into CA path, setting up Gitweb and SourceGitweb plugin integration is working.
I had to abandon SourceCgit plugin as it is still throwing errors...
All in all seems all this procedure is simple when you are guided through it.
Thank you Starbuck for your help in this thread.

So, to sum up my case I had to:
- install php curl package
- add self-signed cert to CA path
- install Gitweb
- install SourceGitweb and abandon SourceCgit
All in few minutes.

mntse
Posts: 7
Joined: Jan 02, 2019 3:11 pm

Re: Git integration using source-integration plugin fails

Post by mntse » Sep 08, 2019 2:49 am

Unfortunately after updating my test install to Mantis 2.21.2 all this stopped working. All the steps which succeed last time now do not work and I am having errors and cannot import git archive anymore. Tried also with latest Source-integration plugin version. I have to revisit this in next weeks.
If anyone here has working setup on production for Gitweb, Cgit or Generic please share your config.

mntse
Posts: 7
Joined: Jan 02, 2019 3:11 pm

Re: Git integration using source-integration plugin fails

Post by mntse » Oct 18, 2019 1:31 pm

Just wanted to report that it seems curl didn't like my selfsigned certificate without alternate hostname embedded. I didn't want to recreate it at the time, so just fixed the link in Mantis Gitweb config. After those curl started to download the contents from the Gitweb site. All good. All works.
I have some integration problems, but this is another story.
EOT.

Post Reply