Bug #132866 for Net-IMAP-Client: Net-IMAP-Client search() method fails against outlook.office365.com imap(s) server

Mon Jun 22 15:14:43 2020 gfb.0x08005bffffff [...] yahoo.com - Ticket created

Subject:	Net-IMAP-Client search() method fails against outlook.office365.com imap(s) server
Date:	Mon, 22 Jun 2020 18:37:30 +0000 (UTC)
To:	"bug-Net-IMAP-Client [...] rt.cpan.org" <bug-net-imap-client [...] rt.cpan.org>
From:	Guy Boyd <gfb.0x08005bffffff [...] yahoo.com>

Apologies for the immediate follow up. I just discovered with a manual dialog using openssl : openssl s_client -connect outlook.office365.com:993 -crlf a001 LOGIN [ username ] [ password ] ? LIST "" "*" # get folder-names, succeeds ? SELECT [ folder-name ] # succeeds #-- response * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent) * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags * OK [UNSEEN 4] Is the first unseen message * OK [UIDVALIDITY 1309] UIDVALIDITY value * OK [UIDNEXT 14833] The next unique identifier value ? OK [READ-WRITE] SELECT completed. # well then. Try SEARCH. ? SEARCH ? BAD Command Argument Error. 12 # Hmm. ? SEARCH UNSEEN # -- response * SEARCH 4 6 7 10 14 15 18 20 (......) # SUCCESS It would seem that office365 wants arguments supplied yo SEARCH that would seem to be optional according to rfc3501, but are probably buried someplace in the polyglot of extension language in rfc5182. I'm no expert at deep RFC corners ( except for abusing project managers with RFC 2119 ), perhaps I've missed something. In any case I'll patch my local use case to unconditionally send the required arguments when the server is MS outlook.office365.com. The module maintainer might maybe consider some creeping elegance (or a pod note) that includes some "office365-ish-extensions" flags if others might find it useful. ( Just a suggestion ).

Mon Jun 22 19:21:46 2020 joelz [...] pobox.com - Correspondence added

Subject:	Re: [rt.cpan.org #132866] Net-IMAP-Client search() method fails against outlook.office365.com imap(s) server
Date:	Mon, 22 Jun 2020 13:21:16 -1000
To:	Guy Boyd via RT <bug-Net-IMAP-Client [...] rt.cpan.org>
From:	Joel Roth <joelz [...] pobox.com>

If I understand correctly, you're saying you get the following results when using imap server outlook.office365.com: $messages = $imap->search() # does not return expected messages $messages = $imap->search('ALL') # does not return expected messages $messages = $imap->search('UNSEEN') # returns messages I'm not sure the first is a problem. The examples for Net::IMAP::Client always indicate supplying arguments to search(). The second seems strange to me, that outlook does not support the SEARCH ALL sent by Net::IMAP::Client. Which additional arguments are needed for search('ALL') to work? Thank you -- Joel Roth

Mon Jun 22 19:21:46 2020 The RT System itself - Status changed from 'new' to 'open'

Tue Jun 23 17:38:38 2020 gfb.0x08005bffffff [...] yahoo.com - Correspondence added

Subject:	Re: [rt.cpan.org #132866] Net-IMAP-Client search() method fails against outlook.office365.com imap(s) server
Date:	Tue, 23 Jun 2020 21:38:24 +0000 (UTC)
To:	<bug-Net-IMAP-Client [...] rt.cpan.org>
From:	Guy Boyd <gfb.0x08005bffffff [...] yahoo.com>

Hi, On Monday, June 22, 2020, 07:21:50 PM EDT, Joel Roth via RT <bug-net-imap-client@rt.cpan.org> wrote: <URL: https://rt.cpan.org/Ticket/Display.html?id=132866 > Show quoted text

> If I understand correctly, you're saying you get the > following results when using imap server > outlook.office365.com:

Show quoted text

> $messages = $imap->search() # does not return expected messages > $messages = $imap->search('ALL') # does not return expected messages > $messages = $imap->search('UNSEEN') # returns messages

Thanks for your response. Sorry for not being clear. I also have additional information that clarifies the problem ( and a hackish local resolution that seems to be working but it is untested for breakage of the original apparently RFC compliant servers ). 1: No SEARCH commands ( where the required arguments are supplied ) return any results (other than undef) via Net::IMAP::Client when executed against my outlook.office365.com session in. No 'non-argument' attempts were made. 2: When debugging the same IMAP commands with openssl s_client -connect [args] I discovered initially that SEARCH UNSEEN works fine from the openssl interface. Later I discovered that SEARCH ALL and SEARCH SEEN and others also appear to work fine from openssl given the new information under 4: below. 3. $summaries = $imap->get_summaries([ @msg_ids ]) was discovered later to "not work" with office365 (after a local patch of 'SEARCH' as described below). 4: ** new information** : What I eventually discovered after several passes is that the office365 apparently does not like the arguments to the SEARCH command: ( $charset ||= 'UTF-8' ) or (or does not like the argument order? untested) I agree that this would seem to violate the RFC if it is true. ( /root/.cpan/build/Net-IMAP-Client-0.9505-nydr2H/lib/Net/IMAP/Client.pm: ) # this is the original installation source. I've taken a saw to the installed module already: in 282 sub search { : 285 $charset ||= 'UTF-8'; and 297 $charset = "CHARSET $charset"; If I remove the charset assertion and send the search on its way , SEARCH starts working. Coming at it backwards at it from the pod, I see this: "Our module is a bit more paranoid and it will actually add charset for both SORT and SEARCH. If $charset is omitted (or undef) the it will default to "UTF-8", which, supposedly, is supported by all IMAP servers". ( I'm really trying to resist saying "except $Msoft") here. 5: ** new information ** I was of course sure I was ready to go, Until $imap->get_summaries() returned undev ( having been passed the required array ref of message ids). From the pod: # fetch message summaries (actually, a lot more) # copy / paste from pod my $summaries = $imap->get_summaries([ @msg_ids ]); I'll need to supply some follow-up debug output on this (i'm in the middle of a functioning pass with a largish mailbox) but the two things I seemed to have discovered are 5a. office365 apparently does not like the original syntax of the commands that use FETCH. I'll send a DIFF of the original syntax and my hacked result later ( a day or so.) Here is my hackish diff for send_cmd that appears to help that. I have not gone back to test what this breaks on other apparently RFC compliant servers, I intent to do that soon. It appears that adding braces around $args in $self->_send_cmd($cmd, [ $args ] ) at sub _tell_imap () things appear to work as expected. (Granted I may have sawed past something in usage that I misunderstood in my haste.) <--- hacked ---> original diff -C 3 /root/perl5/lib/perl5/Net/IMAP/Client.pm /root/.cpan/build/Net-IMAP-Client-0.9505-nydr2H/lib/Net/IMAP/Client.pm sub _tell_imap { ! my ($self, $cmd, $args, $do_notf ) = @_; $cmd = uc $cmd; my ($lineparts, $ok, $res); - # printf("%s %s %s\n", "tell_imap()", $cmd, Data::Dumper::Dumper ($args)); - - if($cmd =~ /FETCH/) { - # sleep 3; - } - RETRY1: { ! $self->_send_cmd($cmd, [ $args ] ); redo RETRY1 if $self->_reconnect_if_needed; } --- 789,802 ---- } sub _tell_imap { ! my ($self, $cmd, $args, $do_notf) = @_; $cmd = uc $cmd; my ($lineparts, $ok, $res); RETRY1: { ! $self->_send_cmd($cmd, $args); redo RETRY1 if $self->_reconnect_if_needed; } Thanks.

Wed Jun 24 13:14:49 2020 gfb.0x08005bffffff [...] yahoo.com - Correspondence added

Subject:	Re: [rt.cpan.org #132866] Net-IMAP-Client search() method fails against outlook.office365.com imap(s) server
Date:	Wed, 24 Jun 2020 17:14:40 +0000 (UTC)
To:	<bug-Net-IMAP-Client [...] rt.cpan.org>
From:	Guy Boyd <gfb.0x08005bffffff [...] yahoo.com>

DEBUG printf of data immediately before it enters the socket. The mods shown here are the minimal required brute-force test to regain search function. This is not IMO an advisable way to implement a patch. Summary conclusion on SEARCH is that office365 does not accept the CHARSET UTF-8 assertion and is further oddly picky about white-space between tokens (apparently). Below, Net::IMAP::Client::_send_cmd () indicates the data written to the socket towards the server. 1: install module on a fresh virtual machine of similar environment. Linux cos7x64-node5-codepool 3.10.0-1127.8.2.el7.x86_64 #1 SMP Tue May 12 16:57:42 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Module < Net::IMAP::Client (GANGLION/Net-IMAP-Client-0.9505.tar.gz) >> /usr/bin/make install -- OK // test code minus sensitive information. $imap->login || die('Login failed: ' . $imap->last_error); $imap->select( 'ACL' ) || die("$!"); my $messages = $imap->search('ALL'); printf("%s: %s\n", "Dumping retval from imap->search('ALL')" , Dumper($messages)); 2: Open a session to outlook.office365.com. Dump return value from $imap->search('ALL'). perl -w imap_test.pl Dumping retval from imap->search('ALL'): $VAR1 = undef; 3: remove charset assertion diff /root/perl5/lib/perl5/Net/IMAP/Client.pm /root/.cpan/build/Net-IMAP-Client-0.9505-4FBM27/lib/Net/IMAP/Client.pm 300c300 < $charset = ''; --- Show quoted text

>

perl -w imap_test.pl Dumping retval from imap->search('ALL'): $VAR1 = undef; 4. Put 'charset' back. Add debug. diff /root/perl5/lib/perl5/Net/IMAP/Client.pm /root/.cpan/build/Net-IMAP-Client-0.9505-4FBM27/lib/Net/IMAP/Client.pm 300c300 < --- Show quoted text

>

681c681 < my($__PACKAGE__,$__FILE__,$__LINE__,$__FUNCTION__) = caller(0); --- Show quoted text

>

702d701 < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); 707d705 < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); Run test. perl -w imap_test.pl Net::IMAP::Client::_send_cmd 800 cmd NIC2 SELECT "ACL" Net::IMAP::Client::_send_cmd 800 cmd NIC3 UID SEARCH CHARSET UTF-8 ALL Dumping retval from imap->search('ALL'): $VAR1 = undef; Net::IMAP::Client::_send_cmd 82 cmd NIC4 LOGOUT 5. Manually remove CHARSET UTF8 ALL immediately prior to socket with a regex. diff /root/perl5/lib/perl5/Net/IMAP/Client.pm /root/.cpan/build/Net-IMAP-Client-0.9505-4FBM27/lib/Net/IMAP/Client.pm 300c300 < --- Show quoted text

>

681c681 < my($__PACKAGE__,$__FILE__,$__LINE__,$__FUNCTION__) = caller(0); --- Show quoted text

>

702,705d701 < $cmd =~ s/CHARSET//g; < $cmd =~ s/UTF-8//g; < $cmd =~ s/\+/ /g; < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); 710d705 < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); Retest. perl -w imap_test.pl Net::IMAP::Client::_send_cmd 803 cmd NIC2 SELECT "ACL" Net::IMAP::Client::_send_cmd 803 cmd NIC3 UID SEARCH ALL Dumping retval from imap->search('ALL'): $VAR1 = undef; Net::IMAP::Client::_send_cmd 82 cmd NIC4 LOGOUT 6. Collapse the extra white-space in the data that was written to the socket. Leave one space behind. $cmd =~ s/\s+ALL/ ALL/g below. 300c300 < --- Show quoted text

>

681c681 < my($__PACKAGE__,$__FILE__,$__LINE__,$__FUNCTION__) = caller(0); --- Show quoted text

>

702,705d701 < $cmd =~ s/CHARSET//g; < $cmd =~ s/UTF-8//g; < $cmd =~ s/\s+ALL/ ALL/g; < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); 710d705 < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); SUCCESS. This seems to support the conclusion that office365 does not either like extraneous white space, OR the CHARSET assertion. perl -w imap_test.pl Net::IMAP::Client::_send_cmd 803 cmd NIC2 SELECT "ACL" Net::IMAP::Client::_send_cmd 803 cmd NIC3 UID SEARCH ALL Dumping retval from imap->search('ALL'): $VAR1 = [ 1, 2, 3, 4, 5, 6, 7, 8, 9 ]; Net::IMAP::Client::_send_cmd 82 cmd NIC4 LOGOUT 7. Put back CHARSET UTF-8 as a final test. diff /root/perl5/lib/perl5/Net/IMAP/Client.pm /root/.cpan/build/Net-IMAP-Client-0.9505-4FBM27/lib/Net/IMAP/Client.pm 300c300 < --- Show quoted text

>

681c681 < my($__PACKAGE__,$__FILE__,$__LINE__,$__FUNCTION__) = caller(0); --- Show quoted text

>

702,705d701 < # $cmd =~ s/CHARSET//g; < # $cmd =~ s/UTF-8//g; < $cmd =~ s/\s+ALL/ ALL/g; < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); 710d705 < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); perl -w imap_test.pl Result: FAILURE. Net::IMAP::Client::_send_cmd 803 cmd NIC2 SELECT "ACL" Net::IMAP::Client::_send_cmd 803 cmd NIC3 UID SEARCH CHARSET UTF-8 ALL Dumping retval from imap->search('ALL'): $VAR1 = undef; Net::IMAP::Client::_send_cmd 82 cmd NIC4 LOGOUT Put back white-space removal. try a SEEN search. diff /root/perl5/lib/perl5/Net/IMAP/Client.pm /root/.cpan/build/Net-IMAP-Client-0.9505-4FBM27/lib/Net/IMAP/Client.pm 300c300 < --- Show quoted text

>

681c681 < my($__PACKAGE__,$__FILE__,$__LINE__,$__FUNCTION__) = caller(0); --- Show quoted text

>

702,706d701 < $cmd =~ s/CHARSET//g; < $cmd =~ s/UTF-8//g; < $cmd =~ s/\s+ALL/ ALL/g; < $cmd =~ s/\s+SEEN/ SEEN/g; < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); 711d705 < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); SUCCESS perl -w imap_test.pl Net::IMAP::Client::_send_cmd 804 cmd NIC2 SELECT "ACL" Net::IMAP::Client::_send_cmd 804 cmd NIC3 UID SEARCH SEEN Dumping retval from imap->search('ALL'): $VAR1 = [ 1, 2, 3, 4, 5, 6, 7, 8, 9 ]; Net::IMAP::Client::_send_cmd 82 cmd NIC4 LOGOUT

Wed Jun 24 13:38:09 2020 gfb.0x08005bffffff [...] yahoo.com - Correspondence added

Subject:	Re: [rt.cpan.org #132866] Net-IMAP-Client search() method fails against outlook.office365.com imap(s) server
Date:	Wed, 24 Jun 2020 17:37:47 +0000 (UTC)
To:	<bug-Net-IMAP-Client [...] rt.cpan.org>
From:	Guy Boyd <gfb.0x08005bffffff [...] yahoo.com>

With respect to FETCH commands it would seem that the minimal required brute-force patch to fix SEARCH no also fixes FETCH. ( for office 365. ) This was unexpected to me, as my own local not-so-minimalist approach to a fix for FETCH seemed to require more changes than this. Today, with no modifications, commands that use FETCH work with no further intervention than reported in the previous message. !diff diff /root/perl5/lib/perl5/Net/IMAP/Client.pm /root/.cpan/build/Net-IMAP-Client-0.9505-4FBM27/lib/Net/IMAP/Client.pm 300c300 < --- Show quoted text

>

681c681 < my($__PACKAGE__,$__FILE__,$__LINE__,$__FUNCTION__) = caller(0); --- Show quoted text

>

702,706d701 < $cmd =~ s/CHARSET//g; < $cmd =~ s/UTF-8//g; < $cmd =~ s/\s+ALL/ ALL/g; < $cmd =~ s/\s+SEEN/ SEEN/g; < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); 711d705 < printf("%s %s %s %s\n", $__FUNCTION__, $__LINE__, 'cmd', $cmd) if ($cmd !~ /LOGIN/); # test code. sensitive info removed $imap->login || die('Login failed: ' . $imap->last_error); $imap->select( 'ACL' ) || die("$!"); my $messages = $imap->search('ALL'); printf("%s: %s\n", "Dumping retval from imap->search('ALL')" , Dumper($messages)); my ( $summaries ) = $imap->get_summaries( $messages ); foreach my $s (@{$summaries}) { printf("%s %s %s\n", $s->{uid}, $s->{date}, $s->rfc822_size ); } Result: SUCCESS: Dumping retval from imap->search('ALL'): $VAR1 = [ 1, 2, 3, 4, 5, 6, 7, 8, 9 ]; Net::IMAP::Client::_send_cmd 804 cmd NIC4 UID FETCH 1,2,3,4,5,6,7,8,9 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODYSTRUCTURE) 1 Thu, 30 Mar 2017 07:13:17 +0000 130872 2 Thu, 30 Mar 2017 07:12:08 +0000 153700 3 Thu, 30 Mar 2017 07:10:49 +0000 118971 4 Thu, 30 Mar 2017 07:09:44 +0000 223423 5 Thu, 30 Mar 2017 07:08:05 +0000 90786 6 Thu, 30 Mar 2017 07:07:04 +0000 128925 7 Thu, 30 Mar 2017 07:05:54 +0000 131248 8 Thu, 30 Mar 2017 07:03:33 +0000 120273 9 Thu, 30 Mar 2017 07:04:44 +0000 120288 Net::IMAP::Client::_send_cmd 82 cmd NIC5 LOGOUT On Monday, June 22, 2020, 07:21:50 PM EDT, Joel Roth via RT <bug-net-imap-client@rt.cpan.org> wrote: <URL: https://rt.cpan.org/Ticket/Display.html?id=132866 > If I understand correctly, you're saying you get the following results when using imap server outlook.office365.com: $messages = $imap->search() # does not return expected messages $messages = $imap->search('ALL') # does not return expected messages $messages = $imap->search('UNSEEN') # returns messages I'm not sure the first is a problem. The examples for Net::IMAP::Client always indicate supplying arguments to search(). The second seems strange to me, that outlook does not support the SEARCH ALL sent by Net::IMAP::Client. Which additional arguments are needed for search('ALL') to work? Thank you -- Joel Roth

Wed Jun 24 13:54:51 2020 gfb.0x08005bffffff [...] yahoo.com - Correspondence added

Subject:	Re: [rt.cpan.org #132866] Net-IMAP-Client search() method fails against outlook.office365.com imap(s) server
Date:	Wed, 24 Jun 2020 17:54:42 +0000 (UTC)
To:	bug-net-imap-client [...] rt.cpan.org
From:	Guy Boyd <gfb.0x08005bffffff [...] yahoo.com>

Final simplistic retest with different (formerly working without change) IMAP server (gmail). Retest against GMAIL: Summary: SUCCESS (no apparent side effects of previous debug hack under simplistic test. # test code 'server' => 'imap.gmail.com', $imap->login || die('Login failed: ' . $imap->last_error); $imap->select( 'juniper_spam' ) || die("$!"); my $messages = $imap->search('ALL'); printf("%s: %s\n", "Dumping retval from imap->search('ALL')" , Dumper($messages)); my ( $summaries ) = $imap->get_summaries( $messages ); foreach my $s (@{$summaries}) { printf("%s %s %s\n", $s->{uid}, $s->{date}, $s->rfc822_size ); } Net::IMAP::Client::_send_cmd 804 cmd NIC2 SELECT "different_folder" Net::IMAP::Client::_send_cmd 804 cmd NIC3 UID SEARCH ALL Dumping retval from imap->search('ALL'): $VAR1 = [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, ]