OAuth2 email authentication method needed

[mushu]

In the email settings screen here http://[server]/plugin.php?page=EmailReporting/manage_mailbox as per the docs here http://www.mantisbt.org/wiki/doku.php/m ... uth_method needs to have the OAuth2 option for authenticating user accounts via POP3 and IMAP to outlook.office365.com for mailboxes "in the cloud". This is becoming more and more common in business and that is a very common authentication method in use nowadays.

[mushu]

EDIT: I found the following links that might be of interest:

MS docs on OAuth authentication:
https://docs.microsoft.com/en-us/exchan ... sing-oauth

PHP OAuth2 code for Microsoft:
https://github.com/stevenmaguire/oauth2-microsoft

MS has stopped supporting "basic" authentication and requires OAuth2 since last year:
https://devblogs.microsoft.com/microsof ... customers/

With so many businesses using Exchange Online and moving to the cloud, the code will need to be updated to keep up. I don't know PHP and don't know how to write Mantis plugins and as the sole webmaster for the place I work I simply couldn't do this. Just putting this out there that a significant number of people who use email for Mantis won't be able to use it any more after a while if the code doesn't keep up with technology. If I can't find a way to authenticate to Exchange Online through Mantis then we will have to move to another ticketing system, and that sucks. :-(

[mushu]

I also found this which might be easier to implement into the code: https://github.com/adoy/PHP-OAuth2

[SL-Gundam]

Microsoft has not stopped basic auth. It allows you to whitelist IP addresses which bypass oauth.

The need is increasing though. But nobody has volunteered time to develop the functionality. So that means it's going to take some time.

I'm sorry i cannot give a better answer.

[francescobianco]

Hi,
please, get you a chance to PHP-IMAP2 (https://github.com/javanile/php-imap2)
This is a FULL implementation of standard PHP IMAP functions like (imap_open, imap_getmailboxes, imap_*, etc...)

This was full tested, every input to imap2_* functions get back the same output of imap_* equivalent

https://github.com/javanile/php-imap2/b ... tyTest.php
The IMAP2 works well with OAUTH

https://github.com/javanile/php-imap2/b ... thTest.php
This libray can be installed with composer

composer require javanile/php-imap2
This libray introduce a easy way to replace the old PHP-IMAP with new one:

JUST replace all imap_(...) functions with imap2_(...)

NO OTHER AMENDS are required.

Please give me the opportunity to make my sacrifices useful to the community.

[SL-Gundam]

EmailReporting uses an older PHP PEAR Net_IMAP package.
So it's not as easy as renaming a couple functions.
But thanks for this. It should help future development.

[mushu]

Microsoft has not stopped basic auth. It allows you to whitelist IP addresses which bypass oauth.

The need is increasing though. But nobody has volunteered time to develop the functionality. So that means it's going to take some time.

I'm sorry i cannot give a better answer.

Except for this: https://docs.microsoft.com/en-us/exchan ... nge-online

When will this change take place?
We've already started making this change. New Microsoft 365 tenants are created with Basic authentication already turned off as they have Security defaults enabled.

Beginning in early 2021, we started to disable Basic authentication for existing tenants with no reported usage. We always provide Message Center notifications to any customer prior to Basic authentication being disabled in their tenant.

In September 2021, we announced that effective October 1, 2022, we will begin disabling Basic authentication for Outlook, EWS, RPS, POP, IMAP, and EAS protocols in Exchange Online.

[gent]

Hi,

are there any developers already working on this? As my company forces me to migrate on premise e-mail server to the microsoft cloud and we're using EmailReporting as core business process, i need a timeframe, when it'll be possible to access Microsoft 365 mailboxes from MantisBT EmailReporting Plugin (and also Mantis core to send e-mails)? I don't want to switch my whole Mantis environment to another product.

Any workarounds for this from people who are suffering from discontinued basic-auth in existing environments?

Please help.

Kind regards

[mushu]

I wrote a little command-line script to log into our ticketing system mailbox using OAuth and grab the raw email data and stuff it into a disk file. Then I changed a bit of the Mantis & EmailReporting code to point to that file and trick it into thinking it is actually online and reading the mailbox content. It works fine. Would *much* rather have the software do it the proper way, but until someone takes the time to update Mantis & EmailReporting code that won't happen. Note that I am using POP3 and not IMAP. Here is the C# program:

Code: Select all

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Graph;
using System.Net;
using System.Net.Http;
using System.Diagnostics;

/*
 * 07/2022
 * Grab unread messages from mailbox and store as individual sequential text files for Mantis Email script to process into tickets because Mantis doesn't know OAuth yet.
 */

 namespace TestSimpleGraphApp
{
    internal class Program
    {
        static void Main(string[] args)
        {
            string fileName = "Message.";

            var clientId = "[GUID here]";

            var tenantId = "[GUID here]";

            var scopes = new[] { "Mail.ReadWrite" };

            
            var credential = new NetworkCredential("emailaddress@yourdomain", "emailaccountpassword");
            
            var graphClient = new GraphServiceClient(new ConsoleAppAuthenticationProvider(scopes, clientId, tenantId, credential));

            //Get the Last Email in the Mailbox
            
            var Messages = graphClient.Me.MailFolders["Inbox"].Messages.Request().Filter("isRead ne true").OrderBy("receivedDateTime").GetAsync().GetAwaiter().GetResult();
            var mm = new Message { IsRead = true };

            //Get MIMEcontent (attachments) from email
            int aa = 0;
            foreach (var msg in Messages)
            {
                { 
                    var fileStream = System.IO.File.Create(fileName + aa.ToString() + ".txt");
                    //if (!msg.IsRead.Value) // commented this out since only unread emails are grabbed in the first place (see .Filter above)
                    //{
                        var MimeContent = graphClient.Me.Messages[Messages[aa].Id].Content.Request().GetAsync().GetAwaiter().GetResult();
                        MimeContent.CopyTo(fileStream);
                        MimeContent.Flush();
                        //graphClient.Me.Messages[Messages[aa].Id].Request().UpdateAsync(mm).GetAwaiter().GetResult(); // set message as read
                        graphClient.Me.Messages[Messages[aa].Id].Request().DeleteAsync().GetAwaiter().GetResult(); // delete the message we just read
                    //}
                    fileStream.Flush();
                    fileStream.Close();
                }
                aa++;
           }
        }
    }

    public class ConsoleAppAuthenticationProvider : IAuthenticationProvider
    {
        private readonly Microsoft.Identity.Client.IPublicClientApplication _app;
        private readonly IEnumerable<string> _scopes;
        private readonly NetworkCredential _networkCredential;

        public ConsoleAppAuthenticationProvider(IEnumerable<string> graphScopes, string clientId, string tenantId, NetworkCredential credential)
        {
            _scopes = graphScopes;
            _app = Microsoft.Identity.Client.PublicClientApplicationBuilder.Create(clientId).WithTenantId(tenantId).Build();
            _networkCredential = credential;
        }

        public Task AuthenticateRequestAsync(HttpRequestMessage request)
        {
            var tokenResult = _app.AcquireTokenByUsernamePassword(_scopes,_networkCredential.UserName,_networkCredential.SecurePassword).ExecuteAsync().Result;
            request.Headers.Add("Authorization", "Bearer " + tokenResult.AccessToken);
            return Task.CompletedTask;
        }
    }
}

Then in the EmailReporting\Core directory I modified mail_api.php at private function process_pop3_mailbox():

Code: Select all

$_argv1 = file_get_contents("EmailFileName.txt",FALSE); # grab filename of email message text
$t_msg = file_get_contents($_argv1,FALSE); # slurp in entire file passed on command line
$t_emailresult = $this->process_single_email( $t_msg ); # changed param from $t_Msg to $t_msg
# commented out the rest of function code

And also changed this at private function process_single_email( $p_i, $p_overwrite_project_id = FALSE ):

Code: Select all

$t_msg = $p_i; # added this
#               $t_msg = $this->getMsg( $p_i );  # commented this out

Finally, in the Scripts directory is this batch file to run the whole thing:

Code: Select all

@echo off
rem 07/2022
rem Grabs all unread inbox emails from mailbox@mydomain
rem and deletes them from inbox and writes them to files.
rem Then launches Mantis email processor to create tickets.
rem EmailFileName.txt holds each message file name for PHP.
rem get into correct directory -- MUST update this as Mantis is updated!!
cd \MantisBT2252\plugins\EmailReporting\scripts
rem test for PHP being available
@"C:\Program Files (x86)\PHP\v7.4\php.exe" --version 2>nul
if errorlevel 1 goto nophp
rem launch email grabber
App\GrabEmailScript.exe
rem messages are now serialized as Message.0.txt  Message.1.txt  etc
rem  so process each one individually
for %%x in (Message.*.txt) do call cmd /c "php bug_report_mail.php %%x"
rem do not delete files if there was any returned error code
if errorlevel 1 goto xit
del Message.*.txt
goto xit
:nophp
echo ** No PHP found?!
:xit

I don't remember if I made any other changes to code elsewhere but this is a start. Needless to say this is a very Windows-oriented solution. But it works so hopefully a real solution will come out before anything here breaks!

[SL-Gundam]

I tried getting this working but it seems that because EmailReporting is still using PEAR Imap packages and not the PHP Imap extension this is problematic.

EmailReporting code first needs to be changed so that it uses the PHP Imap extension. After that support can be added for OAuth.

So i'm sorry this is gonna take longer then hoped. Anybody with experience with Imap and OAuth is welcome to help.

[mushu]

Well, except that I can't use IMAP for this application, it must be POP3 so it would be great to ensure that both protocols are allowed in whatever solution is decided upon.

[sambuca]

I have created a version where I have changed out the imap part and added oauth capability: https://github.com/magnebra/EmailReport ... ap_library

First off, php is not my primary programming language so there may be mistakes or bad practices all over the code.

I have tried to avoid changing too much in the core EmailReporting plugin and instead created a small IMAP.php class that uses similar functions to the old Net_Imap code. This new IMAP.php class uses the javanile/php-imap2 library mentioned previously in this thread.

It also uses a library for oauth2; TheNetworg/oauth2-azure
This is based on https://github.com/thephpleague/oauth2-client, which has support for different oauth2 providers. Currently the code is hardcoded to use the azure provider.

Configuration options are added to the mailbox edit screen to set oauth parameters.

For reference I've used the description from Microsoft on how to set up this https://learn.microsoft.com/en-us/excha ... onnections using the client credentials grant flow.

Hope this helps someone.

[love2scoot]

@sambuca:

I'm not able to get this working. Attempting to run bug_report_mail.php from the command line results in:

Code: Select all

syntax error, unexpected 'private' (T_PRIVATE)
/var/www/mantis/myinstance/plugins/EmailReporting/pages/bug_report_mail.php: 46: - - - - plugin_require_api( <string>'core/mail_api.php' )
/var/www/mantis/myinstance/plugin.php: 74: - - - - include( <string>'/var/www/mantis/myinstance/plugins/EmailReporting/pages/bug_report_mail.php' )
/var/www/mantis/myinstance/plugins/EmailReporting/scripts/bug_report_mail.php: 28: - - - - require_once( <string>'/var/www/mantis/myinstance/plugin.php' )

This is a newly installed server where I have migrated the data from an existing instance in production for testing purposes:
Ub18.04LTSx64 Server
Mantis v2.25.5
Composer 1.6.3 (This is not installed by default)

Using the EmailReporting plugin with a mailbox on Microsoft365 configured for IMAP using "USER" authentication:
v0.10.1 works fine - reports "Done checking all mailboxes"
v0.11.0-DEV works fine - reports "Done checking all mailboxes"
v0.11.0-OAUTH (your branch) shows the error above

(Note that I managed to push off closure of basic IMAP authentication until 2023-01-01, thus the "USER" authentication method still works for my tenant, at least for now)

I also tried using the "XOAUTH2" authentication method (after registration of the application on Azure) and population of the Client id, Tenant, and Scopes fields in the "Manage Mailboxes" interface, but I still get the same error with bug_report_mail.php (although the settings are correctly being set in the GUI).

Are there unmet dependencies here? Typically MantisBT plugins are either self contained or include a dependency note for another plugin, but I don't see anything beyond the MantisBT Core plugin.

Also, thanks for putting in the work here - I'd love to continue using MantisBT for bug tracking, but this OAUTH2 limitation on the EmailReporting plugin could be a breaking issue for me

[sambuca]

It seems I had deleted some code by accident. Perhaps that was the root of your problem.

The dependencies should all be installed by running composer in the root folder of the plugin.

[love2scoot]

Thanks @sambuca, almost there.

For reference, I had to walk through the following debug:

• After updating to the latest commit, I attempted to manually run

  Code: Select all

  /usr/bin/php /var/www/mantis/myinstance/plugins/EmailReporting/scripts/bug_report_mail.php

  to check for email. This produced an error stating the the folder /var/www/mantis/myinstance/plugins/EmailReporting/vendor was missing.
• Some quick Googling suggested that I needed to run

  Code: Select all

  composer install

  in the root ot the EmailReporting folder.
• When attempting this, it failed stating that the prerequisite PHPv7.4 was not present (PHPv7.4 was available starting at Ubuntu LTS v20.04). For Ubuntu LTS v18.04, I followed the install guide for installing from PPA.
• Once PHPv7.4 was installed, the composer install command worked as expected and the /vendor folder appeared.
• Finally, attempting to run

  Code: Select all

  /usr/bin/php /var/www/mantis/myinstance/plugins/EmailReporting/scripts/bug_report_mail.php

  appeared to work (functionally) but now I'm working through errors on the authentication side.

How did you populate the XOAUTH2 fields in the Mailbox Settings screen? My approach was:

1. Log into my tenant and Navigate to Azure Active Directory / App registrations to register the MantisBT instance
2. Once registered, I could open the app and I used the following fields:
   • MantisBT Mailbox OAUTH2 only settings Client ID field: Application (client) ID
   • MantisBT Mailbox OAUTH2 only settings Redirect uri field: Blank
   • MantisBT Mailbox OAUTH2 only settings Tenant field: Directory (tenant) ID
   • MantisBT Mailbox OAUTH2 only settings Scopes field: https://outlook.office.com/IMAP.AccessA ... l/.default

When attempting to run

Code: Select all

bug_report_mail.php

now I get the error:

Code: Select all

AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app 'My App uid'

I attempted to manually add a secret to the application in the Azure Active Directory / App registrations window, but I'm not sure where to put the Secret value.

Any insight?