OAuth2 email authentication method needed

This plugin allows you to report an issue in MantisBT by sending an email to a particular mail account

Moderators: Developer, Contributor

stevenrlp
Posts: 4
Joined: 28 Oct 2018, 21:21

Re: OAuth2 email authentication method needed

Post by stevenrlp »

Looking at the code it seems the password field is being used as the secret so you should add the secret into the password field. I have tried for hours to get this to work and the only error I get back is INTERNAL APPLICATION ERROR. I'm guessing my issue is due to the port on the server being blocked but I can't get it unblocked easy without lots of emails. I was using POP before on 995, IMAP uses 993 so I guess that's my issue. I hope someone can get it to work for POP also.
sambuca
Posts: 3
Joined: 09 Nov 2022, 08:04

Re: OAuth2 email authentication method needed

Post by sambuca »

Sorry, I should have provided some more information about the setup. It seems that the MS document does not match exactly what I have setup up currently.

So, the password field is the client secret, so that must be filled in.

You also need to fill in the four fields under OAUTH2:
Client ID - application id from Azure AD
Redirect Uri - not used, but I believe this must match one of the configured redirect uris for the application in AAD
Tenant: directory tenant id
Scopes: I currently have https://outlook.office365.com/.default which works for me.

Hope this helps.
v_wagner
Posts: 3
Joined: 11 Jan 2023, 09:19

Re: OAuth2 email authentication method needed

Post by v_wagner »

Hi,
it's a really important and very useful extension, but I can't get it to work. Sometimes I program, so I can monitor what's going on inside the code, but I still haven't been able to connect it to the office365.com mailbox.

I went through the installation, adding libraries via composer, as well as adding the application and secret code in Azure. I believe that everything is set correctly, but the result is always the same

[pear_error: message="AUTHENTICATE XOAUTH2: A0001 NO AUTHENTICATE failed., host = outlook.office365.com, port = 993, auth-meth: XOAUTH2" code=0 mode=return level=notice prefix="" info="" ]

I was trying to debug the login process and before I set the scopes to the correct version, I was finding the wrong url scope message in the login procedure (core_pear/IMAP.php) in $pass instead of the token. Now there is a long base64 string, which should be OK.

A subsequent call to connect then returns an empty value to $ret.
The input to connect is then this:

[$this->host] => outlook.office365.com
[$user] => testdesk@***** (user)
[$pass] => eyJ******PFpxg (token)
[array] => Array
(
[port] => 993
[timeout] => 3
[ssl_mode] => tlsv1.2
[auth_type] => XOAUTH2
[force_caps] =>
)

In the connect function, it passes the tests up to the authenticate call, internally up to the branch

else if ($type == 'XOAUTH2') {

Here, the username is combined with the token and recoded to Base64 once more, stored in the $auth variable, and then the value "A0001 NO AUTHENTICATE failed" is stored. to $line and returns -1 in the result.

I've already spent over 20 hours on this and I must be doing something wrong somewhere.

The setting is

["enabled"]=> 1
["description"]=> "TestDesk"
["mailbox_type"]=> "IMAP"
["hostname"]=> "tlsv1.2://outlook.office365.com"
["port"]=> 993
["encryption"]=> "TLSv1.2"
["ssl_cert_verify"]=> 0/1 (same behavior)
["erp_username"]=>"testdesk@ ***domain*****"
["erp_password"]=> string(56) "****secret code from Azure*****"
["auth_method"]=> string(7) "XOAUTH2"
["erp_clientid"]=> string(36) "a17***client ID from Azure****c47"
["erp_redirecturi"]=> ""
["erp_tenant"]=> "abd***tenant ID from Azure***b65"
["erp_scopes"]=>"https://outlook.office365.com/.default"
["imap_basefolder"]=>""
["imap_createfolderstructure"]=>0

However, I have tried all possible combinations of settings and the result is always the same. Only with POP3 is the message different and refers to an empty greeting message.
dcal
Posts: 2
Joined: 11 Jan 2023, 20:16

Re: OAuth2 email authentication method needed

Post by dcal »

v_wagner wrote: 11 Jan 2023, 09:28 Hi,
it's a really important and very useful extension, but I can't get it to work. Sometimes I program, so I can monitor what's going on inside the code, but I still haven't been able to connect it to the office365.com mailbox.

I went through the installation, adding libraries via composer, as well as adding the application and secret code in Azure. I believe that everything is set correctly, but the result is always the same

[pear_error: message="AUTHENTICATE XOAUTH2: A0001 NO AUTHENTICATE failed., host = outlook.office365.com, port = 993, auth-meth: XOAUTH2" code=0 mode=return level=notice prefix="" info="" ]

I was trying to debug the login process and before I set the scopes to the correct version, I was finding the wrong url scope message in the login procedure (core_pear/IMAP.php) in $pass instead of the token. Now there is a long base64 string, which should be OK.

A subsequent call to connect then returns an empty value to $ret.
The input to connect is then this:

[$this->host] => outlook.office365.com
[$user] => testdesk@***** (user)
[$pass] => eyJ******PFpxg (token)
[array] => Array
(
[port] => 993
[timeout] => 3
[ssl_mode] => tlsv1.2
[auth_type] => XOAUTH2
[force_caps] =>
)

In the connect function, it passes the tests up to the authenticate call, internally up to the branch

else if ($type == 'XOAUTH2') {

Here, the username is combined with the token and recoded to Base64 once more, stored in the $auth variable, and then the value "A0001 NO AUTHENTICATE failed" is stored. to $line and returns -1 in the result.

I've already spent over 20 hours on this and I must be doing something wrong somewhere.

The setting is

["enabled"]=> 1
["description"]=> "TestDesk"
["mailbox_type"]=> "IMAP"
["hostname"]=> "tlsv1.2://outlook.office365.com"
["port"]=> 993
["encryption"]=> "TLSv1.2"
["ssl_cert_verify"]=> 0/1 (same behavior)
["erp_username"]=>"testdesk@ ***domain*****"
["erp_password"]=> string(56) "****secret code from Azure*****"
["auth_method"]=> string(7) "XOAUTH2"
["erp_clientid"]=> string(36) "a17***client ID from Azure****c47"
["erp_redirecturi"]=> ""
["erp_tenant"]=> "abd***tenant ID from Azure***b65"
["erp_scopes"]=>"https://outlook.office365.com/.default"
["imap_basefolder"]=>""
["imap_createfolderstructure"]=>0

However, I have tried all possible combinations of settings and the result is always the same. Only with POP3 is the message different and refers to an empty greeting message.
I'm also trying to use sambuca's fork, and also getting the exact same response: "A0001 NO AUTHENTICATE failed" you're getting, with the settings pretty much the same as yours.

I suspect it may be a problem with the Azure AD configuration, I'm consulting my systems administrator tomorrow to go over the Microsoft docs with them: https://learn.microsoft.com/en-us/excha ... onnections to see if I can spot the problem.
Last edited by dcal on 11 Jan 2023, 20:28, edited 1 time in total.
dcal
Posts: 2
Joined: 11 Jan 2023, 20:16

Re: OAuth2 email authentication method needed

Post by dcal »

In relation to my previous post, after reviewing Azure AD with my system's administrator, it now works.

This section of the Microsoft doc is the key:
https://learn.microsoft.com/en-us/excha ... onnections

ALSO when creating the ServicePrincipal, the ObjectId must be that of the Enterprise Application, not the Application Registration. They are different sections in the Azure AD site menu.

This image helped us identify which ObjectId was the correct one:

Image

Once AD is correctly configured, you should get a A0001 OK AUTHENTICATE completed instead of AUTHENTICATE XOAUTH2: A0001 NO AUTHENTICATE failed
v_wagner
Posts: 3
Joined: 11 Jan 2023, 09:19

Re: OAuth2 email authentication method needed

Post by v_wagner »

After configuring the ServicePrincipals correctly, it finally got up and running. Thanks. So there is still some error when terminating the connection - maybe some change in the IMAP library and a non-existent function, so it won't log out correctly, but new tasks will start it.

[23-Jan-2023 09:01:42 Europe/Berlin] Call to undefined method IMAP::cmdExpunge()
D:\xampp\htdocs\mantisbt2\plugins\EmailReporting\core\mail_api.php: 505: IMAP - -> - disconnect( <boolean>true )
D:\xampp\htdocs\mantisbt2\plugins\EmailReporting\core\mail_api.php: 251: ERP_mailbox_api - -> - process_imap_mailbox()
D:\xampp\htdocs\mantisbt2\plugins\EmailReporting\pages\bug_report_mail.php: 74: ERP_mailbox_api - -> - process_mailbox( <array> { ['enabled'] => 1, ['description'] => 'TestDesk', ['mailbox_type'] => 'IMAP', ['hostname'] => 'outlook.office365.com', ['port'] => '', ['encryption'] => 'TLSv1.2', ['ssl_cert_verify'] => 0, ['erp_username'] => 'testdesk@************', ['erp_password'] => 'Nj************g==', ['auth_method'] => 'XOAUTH2', ['erp_clientid'] => 'a17************47', ['erp_redirecturi'] => '', ['erp_tenant'] => 'ab************65', ['erp_scopes'] => 'https://outlook.office365.com/.default', ['project_id'] => 1, ['global_category_id'] => 1, ['imap_basefolder'] => 'INBOX', ['imap_createfolderstructure'] => 0 } )
D:\xampp\htdocs\mantisbt2\plugin.php: 74: - - - - include( <string>'D:\\xampp\\htdocs\\mantisbt2\\plugins\\EmailReporting\\pages\\bug_report_mail.php' )
D:\xampp\htdocs\mantisbt2\plugins\EmailReporting\scripts\bug_report_mail.php: 28: - - - - require_once( <string>'D:\\xampp\\htdocs\\mantisbt2\\plugin.php' )
v_wagner
Posts: 3
Joined: 11 Jan 2023, 09:19

Re: OAuth2 email authentication method needed

Post by v_wagner »

Grrrrrrr - it is produced by this function:

Delete processed email from the mailbox
mushu
Posts: 360
Joined: 04 Jan 2017, 17:41

Re: OAuth2 email authentication method needed

Post by mushu »

Will this be working for POP3 hopefully?
xcesiv
Posts: 4
Joined: 18 Apr 2023, 15:26

Re: OAuth2 email authentication method needed

Post by xcesiv »

Is there still no one that is willing to work on a fix? The company I work for uses Mantis and this is one of their big issues. I've done some work on this from sambuca's fork, just curious if there is any active development now that basic authentication is unavailable from Microsoft?

What sort of obstacles are there in adding this feature? From the outside looking in it doesn't seem overly complicated so maybe I'm a bit naive on the topic.

Will gladly provide assistance to get the plugin working
SL-Gundam
Posts: 728
Joined: 06 Jul 2011, 14:17

Re: OAuth2 email authentication method needed

Post by SL-Gundam »

Besides the work seen here. I don't think anyone is working on it.
I've tried a couple months ago but got pushed into other projects.
The biggest problem as far as i can see is that every company like Microsoft and Google that use Oauth2, put there own spin on it. This makes everything a pain to develop.
xcesiv
Posts: 4
Joined: 18 Apr 2023, 15:26

Re: OAuth2 email authentication method needed

Post by xcesiv »

Yes it seems quite annoying.



Working with Sambuca's fork it seems like a few others have had this issue where their setup looks like this:

Code: Select all

array(18) {
  ["enabled"]=>
  int(1)
  ["description"]=>
  string(16) "test-oauth-edits"
  ["mailbox_type"]=>
  string(4) "IMAP"
  ["hostname"]=>
  string(31) "tlsv1.2://outlook.office365.com"
  ["port"]=>
  int(993)
  ["encryption"]=>
  string(7) "TLSv1.2"
  ["ssl_cert_verify"]=>
  int(1)
  ["erp_username"]=>
  string(17) "[email-address]"
  ["erp_password"]=>
  string(56) "[client-secret]"
  ["auth_method"]=>
  string(7) "XOAUTH2"
  ["erp_clientid"]=>
  string(36) "[client-id]"
  ["erp_redirecturi"]=>
  string(0) ""
  ["erp_tenant"]=>
  string(36) "[tennat-id]"
  ["erp_scopes"]=>
  string(38) "https://outlook.office365.com/.default"
  ["project_id"]=>
  int(25)
  ["global_category_id"]=>
  int(1)
  ["imap_basefolder"]=>
  string(0) ""
  ["imap_createfolderstructure"]=>
  int(0)
}

I replaced the private information here in brackets obviously, but the only error I receive is "INTERNAL APPLICATION ERROR" with no details which makes me inherently think it's an Azure setup related problem?

I do see this error:

Connection refused for URI https://login.microsoftonline.com/[tenant-id]/v2.0/.well-known/openid-configuration?appid=[client-id]

Are your api permissions set with IMAP.AccessAsApp or IMAP.AccessAsUser.All?

I'll go through and try and play with the settings. I think it'd be great if sambouca or someone else who has gotten this fork to successfully work could show exactly what their setup in Azure was looking like to do so too.
Last edited by xcesiv on 19 Apr 2023, 14:12, edited 2 times in total.
xcesiv
Posts: 4
Joined: 18 Apr 2023, 15:26

Re: OAuth2 email authentication method needed

Post by xcesiv »

mushu wrote: 10 Oct 2022, 15:37 I wrote a little command-line script to log into our ticketing system mailbox using OAuth and grab the raw email data and stuff it into a disk file. Then I changed a bit of the Mantis & EmailReporting code to point to that file and trick it into thinking it is actually online and reading the mailbox content. It works fine. Would *much* rather have the software do it the proper way, but until someone takes the time to update Mantis & EmailReporting code that won't happen. Note that I am using POP3 and not IMAP. Here is the C# program:

Code: Select all

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Graph;
using System.Net;
using System.Net.Http;
using System.Diagnostics;

/*
 * 07/2022
 * Grab unread messages from mailbox and store as individual sequential text files for Mantis Email script to process into tickets because Mantis doesn't know OAuth yet.
 */

 namespace TestSimpleGraphApp
{
    internal class Program
    {
        static void Main(string[] args)
        {
            string fileName = "Message.";

            var clientId = "[GUID here]";

            var tenantId = "[GUID here]";

            var scopes = new[] { "Mail.ReadWrite" };

            
            var credential = new NetworkCredential("emailaddress@yourdomain", "emailaccountpassword");
            
            var graphClient = new GraphServiceClient(new ConsoleAppAuthenticationProvider(scopes, clientId, tenantId, credential));

            //Get the Last Email in the Mailbox
            
            var Messages = graphClient.Me.MailFolders["Inbox"].Messages.Request().Filter("isRead ne true").OrderBy("receivedDateTime").GetAsync().GetAwaiter().GetResult();
            var mm = new Message { IsRead = true };

            //Get MIMEcontent (attachments) from email
            int aa = 0;
            foreach (var msg in Messages)
            {
                { 
                    var fileStream = System.IO.File.Create(fileName + aa.ToString() + ".txt");
                    //if (!msg.IsRead.Value) // commented this out since only unread emails are grabbed in the first place (see .Filter above)
                    //{
                        var MimeContent = graphClient.Me.Messages[Messages[aa].Id].Content.Request().GetAsync().GetAwaiter().GetResult();
                        MimeContent.CopyTo(fileStream);
                        MimeContent.Flush();
                        //graphClient.Me.Messages[Messages[aa].Id].Request().UpdateAsync(mm).GetAwaiter().GetResult(); // set message as read
                        graphClient.Me.Messages[Messages[aa].Id].Request().DeleteAsync().GetAwaiter().GetResult(); // delete the message we just read
                    //}
                    fileStream.Flush();
                    fileStream.Close();
                }
                aa++;
           }
        }
    }

    public class ConsoleAppAuthenticationProvider : IAuthenticationProvider
    {
        private readonly Microsoft.Identity.Client.IPublicClientApplication _app;
        private readonly IEnumerable<string> _scopes;
        private readonly NetworkCredential _networkCredential;

        public ConsoleAppAuthenticationProvider(IEnumerable<string> graphScopes, string clientId, string tenantId, NetworkCredential credential)
        {
            _scopes = graphScopes;
            _app = Microsoft.Identity.Client.PublicClientApplicationBuilder.Create(clientId).WithTenantId(tenantId).Build();
            _networkCredential = credential;
        }

        public Task AuthenticateRequestAsync(HttpRequestMessage request)
        {
            var tokenResult = _app.AcquireTokenByUsernamePassword(_scopes,_networkCredential.UserName,_networkCredential.SecurePassword).ExecuteAsync().Result;
            request.Headers.Add("Authorization", "Bearer " + tokenResult.AccessToken);
            return Task.CompletedTask;
        }
    }
}
Then in the EmailReporting\Core directory I modified mail_api.php at private function process_pop3_mailbox():

Code: Select all

$_argv1 = file_get_contents("EmailFileName.txt",FALSE); # grab filename of email message text
$t_msg = file_get_contents($_argv1,FALSE); # slurp in entire file passed on command line
$t_emailresult = $this->process_single_email( $t_msg ); # changed param from $t_Msg to $t_msg
# commented out the rest of function code
And also changed this at private function process_single_email( $p_i, $p_overwrite_project_id = FALSE ):

Code: Select all

$t_msg = $p_i; # added this
#               $t_msg = $this->getMsg( $p_i );  # commented this out
Finally, in the Scripts directory is this batch file to run the whole thing:

Code: Select all

@echo off
rem 07/2022
rem Grabs all unread inbox emails from mailbox@mydomain
rem and deletes them from inbox and writes them to files.
rem Then launches Mantis email processor to create tickets.
rem EmailFileName.txt holds each message file name for PHP.
rem get into correct directory -- MUST update this as Mantis is updated!!
cd \MantisBT2252\plugins\EmailReporting\scripts
rem test for PHP being available
@"C:\Program Files (x86)\PHP\v7.4\php.exe" --version 2>nul
if errorlevel 1 goto nophp
rem launch email grabber
App\GrabEmailScript.exe
rem messages are now serialized as Message.0.txt  Message.1.txt  etc
rem  so process each one individually
for %%x in (Message.*.txt) do call cmd /c "php bug_report_mail.php %%x"
rem do not delete files if there was any returned error code
if errorlevel 1 goto xit
del Message.*.txt
goto xit
:nophp
echo ** No PHP found?!
:xit
I don't remember if I made any other changes to code elsewhere but this is a start. Needless to say this is a very Windows-oriented solution. But it works so hopefully a real solution will come out before anything here breaks!

Curious to try this solution too. @mushu is this how you're still accomplishing this with POP?
mushu
Posts: 360
Joined: 04 Jan 2017, 17:41

Re: OAuth2 email authentication method needed

Post by mushu »

xcesiv wrote: 19 Apr 2023, 13:30 Curious to try this solution too. @mushu is this how you're still accomplishing this with POP?
Yes, still working perfectly. Task Manager on the server is set to launch the batch file every 5 minutes.
xcesiv
Posts: 4
Joined: 18 Apr 2023, 15:26

Re: OAuth2 email authentication method needed

Post by xcesiv »

mushu wrote: 19 Apr 2023, 16:02
xcesiv wrote: 19 Apr 2023, 13:30 Curious to try this solution too. @mushu is this how you're still accomplishing this with POP?
Yes, still working perfectly. Task Manager on the server is set to launch the batch file every 5 minutes.
So does your process_pop3_mailbox function look like this:

Code: Select all

private function process_pop3_mailbox()
	{
		$_argv1 = file_get_contents("EmailFileName.txt",FALSE); # grab filename of email message text
		$t_msg = file_get_contents($_argv1,FALSE); # slurp in entire file passed on command line
		$t_emailresult = $this->process_single_email( $t_msg ); # changed param from $t_Msg to $t_msg
		# COMMENT OUT THE REST OF THE FUNCTION CODE
		/* $this->_mailserver = new Net_POP3();
		$this->_mailserver->_timeout = 3;

		$t_connectresult = $this->_mailserver->connect( $this->_mailbox[ 'hostname' ], $this->_mailbox[ 'port' ], $this->get_StreamContextOptions() );

		if ( $t_connectresult === TRUE )
		{
			$t_loginresult = $this->mailbox_login();

			if ( !$this->pear_error( 'Attempt login', $t_loginresult ) )
			{
				if ( $this->_test_only === FALSE )
				{
					if ( project_get_field( $this->_mailbox[ 'project_id' ], 'enabled' ) == ON )
					{
						$t_ListMsgs = $this->getListing();

						if ( !$this->pear_error( 'Retrieve list of messages', $t_ListMsgs ) )
						{
							while ( $t_Msg = array_pop( $t_ListMsgs ) )
							{
								$t_emailresult = $this->process_single_email( $t_Msg[ 'msg_id' ] );

								if ( $this->_mail_delete && $t_emailresult )
								{
									$t_deleteresult = $this->_mailserver->deleteMsg( $t_Msg[ 'msg_id' ] );

									$this->pear_error( 'Attempt delete email', $t_deleteresult );
								}
							}
						}
					}
					else
					{
						$this->custom_error( 'Project is disabled: ' . project_get_field( $this->_mailbox[ 'project_id' ], 'name' ) );
					}
				}
			}

			$this->_mailserver->disconnect();
		}
		else
		{
			$this->custom_error( 'Failed to connect to the mail server' . ( ( $this->_mailbox[ 'encryption' ] !== 'None' && $this->_mailbox[ 'ssl_cert_verify' ] == ON ) ? '. This could possibly be because SSL certificate verification failed' : NULL ) );
		} */
	}
	
	
mushu
Posts: 360
Joined: 04 Jan 2017, 17:41

Re: OAuth2 email authentication method needed

Post by mushu »

Correct, your code looks perfect.
Post Reply