Bug #117033 for Mail-IMAPClient: accessing messages on an IMAP server by message number

Thu Aug 18 06:29:51 2016 hw [...] gc-24.de - Ticket created

Subject:	accessing messages on an IMAP server by message number
Date:	Thu, 18 Aug 2016 12:29:36 +0200
To:	Perl Beginners <beginners [...] perl.org>, bug-Mail-IMAPClient [...] rt.cpan.org
From:	hw <hw [...] gc-24.de>

Hi, how can I access messages residing on an IMAP server by their message numbers? "Message number" means what MUAs like the one built into Seamonkey display as 'Order received'. The IMAP server being used is cyrus, and the message number corresponds to the name of the file the message is stored in. IIRC, these numbers never change unless you run some program that comes with cyrus to re-number the messages. According to the documentation, Mail::IMAPClient[1] can use message numbers to access messages, but it does not work in that it accesses messages that have different numbers than the number given as parameter to a function Mail::IMAPClient provides. For example: $imap->migrate($targetaccount, 25, $targetfolder); ... copies *a* message (apparently a randomly selected one), but not the message that has the number 25. I have no good way to identify messages other their file names and thus the message numbers derived from them because the virus scanner gives me the file names. I can not simply delete the files because that would screw up the indices and/or whatever cyrus uses to manage the messages. Since it´s not feasible to tell the users that they need to delete a message with a particular number, I´ve written a program to delete (quarantine, actually) the messages the virus scanner thinks contain viruses. It turns out that my program does not work because messages are quarantined apparently randomly, and the messages that contain the viruses remain untouched. I can even run it several times to quarantine the same messages, and it will quarantine another message every time, though the message numbers used are the same each time :( So either Mail::IMAPClient is buggy, or it uses some other, useless way to enumerate the messages by itself rather than using the message numbers the IMAP server uses. (And yes, the UId is set to 0, and I verified that it remains 0.)

Thu Aug 18 06:39:27 2016 hw [...] gc-24.de - Correspondence added

CC:	Bugs in Mail-IMAPClient via RT <bug-Mail-IMAPClient [...] rt.cpan.org>
Subject:	Re: accessing messages on an IMAP server by message number [rt.cpan.org #117033]
Date:	Thu, 18 Aug 2016 12:39:13 +0200
To:	Perl Beginners <beginners [...] perl.org>
From:	hw <hw [...] gc-24.de>

PS: I forgot to add: [1]: http://search.cpan.org/~plobbes/Mail-IMAPClient-3.38/lib/Mail/IMAPClient.pod And the ticket is: https://rt.cpan.org/Ticket/Display.html?id=117033 hw schrieb: Show quoted text

> > Hi, > > how can I access messages residing on an IMAP server by their > message numbers? "Message number" means what MUAs like the > one built into Seamonkey display as 'Order received'. > > The IMAP server being used is cyrus, and the message number > corresponds to the name of the file the message is stored in. > IIRC, these numbers never change unless you run some program > that comes with cyrus to re-number the messages. > > According to the documentation, Mail::IMAPClient[1] can use > message numbers to access messages, but it does not work in that > it accesses messages that have different numbers than the number > given as parameter to a function Mail::IMAPClient provides. > > For example: > > > $imap->migrate($targetaccount, 25, $targetfolder); > > > ... copies *a* message (apparently a randomly selected one), but > not the message that has the number 25. > > I have no good way to identify messages other their file names and > thus the message numbers derived from them because the virus scanner > gives me the file names. I can not simply delete the files because > that would screw up the indices and/or whatever cyrus uses to manage > the messages. > > Since it´s not feasible to tell the users that they need to delete a > message with a particular number, I´ve written a program to delete > (quarantine, actually) the messages the virus scanner thinks contain > viruses. It turns out that my program does not work because messages > are quarantined apparently randomly, and the messages that contain the > viruses remain untouched. I can even run it several times to quarantine > the same messages, and it will quarantine another message every time, > though the message numbers used are the same each time :( > > So either Mail::IMAPClient is buggy, or it uses some other, useless way > to enumerate the messages by itself rather than using the message numbers > the IMAP server uses. > > (And yes, the UId is set to 0, and I verified that it remains 0.) >

Mon Nov 14 10:35:50 2016 PLOBBES [...] cpan.org - Correspondence added

I'm afraid the questions aren't really within the scope of Mail::IMAPClient per se. If you really want to understand what is going on here, then it would probably be worthwhile reading and comprehending the IMAP RFC: https://tools.ietf.org/html/rfc3501 In particular, you may find details on message numbers useful: https://tools.ietf.org/html/rfc3501#section-2.3.1 And then studying the code of whatever MUA (e.g. Seamonkey) you are using to see if the number they display has any correlation with the UID or message sequence numbers. I wouldn't be surprised if they have no direct/obvious correlation - without looking at the code anyway. In addition, then you would likely also want to study the code for the back-end IMAP server and see if they also guarantee that the UID/sequence number corresponds to what you're trying to get at. However, after doing all of that what you are more likely to find is that the "message number" you are referring to in the MUA has little, or possibly no, direct meaning/correlation with the IMAP rfc nor with the IMAP server implementation that you are using. Sorry if this isn't what you want to hear, but that my gut feeling on this request. In the end, if you can determine the appropriate rules, then Mail::IMAPClient may be able to help you make programmatic changes to the mailboxes you wish to modify. Determining if it is trivial, or not, to map those "message numbers" from the client to the UID/sequence numbers that the IMAP server understands is something you'll have to figure out though, sorry.

Mon Nov 14 10:35:51 2016 The RT System itself - Status changed from 'new' to 'open'

Mon Nov 14 10:36:01 2016 PLOBBES [...] cpan.org - Status changed from 'open' to 'rejected'

Mon Nov 14 11:23:21 2016 hw [...] gc-24.de - Correspondence added

Subject:	Re: [rt.cpan.org #117033] accessing messages on an IMAP server by message number
Date:	Mon, 14 Nov 2016 17:22:59 +0100
To:	bug-Mail-IMAPClient [...] rt.cpan.org
From:	hw <hw [...] gc-24.de>

Phil Pearl (Lobbes) via RT schrieb: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=117033 > > > I'm afraid the questions aren't really within the scope of Mail::IMAPClient per se. > > If you really want to understand what is going on here, then it would probably be worthwhile reading and comprehending the IMAP RFC: > https://tools.ietf.org/html/rfc3501 > > In particular, you may find details on message numbers useful: > https://tools.ietf.org/html/rfc3501#section-2.3.1

Thank you very much for the pointers! Show quoted text

> And then studying the code of whatever MUA (e.g. Seamonkey) you are using to see if the number they display has any correlation with the UID or message sequence numbers. I wouldn't be surprised if they have no direct/obvious correlation - without looking at the code anyway. > > In addition, then you would likely also want to study the code for the back-end IMAP server and see if they also guarantee that the UID/sequence number corresponds to what you're trying to get at. > > However, after doing all of that what you are more likely to find is that the "message number" you are referring to in the MUA has little, or possibly no, direct meaning/correlation with the IMAP rfc nor with the IMAP server implementation that you are using. > > Sorry if this isn't what you want to hear, but that my gut feeling on this request. In the end, if you can determine the appropriate rules, then Mail::IMAPClient may be able to help you make programmatic changes to the mailboxes you wish to modify. Determining if it is trivial, or not, to map those "message numbers" from the client to the UID/sequence numbers that the IMAP server understands is something you'll have to figure out though, sorry.

The message numbers I am referring to are the numbers within the file names cyrus uses to store the messages on disk. The message numbers are extracted from a clamav scanning report by a tool I have written to automatically quarantine messages containing viruses via IMAP. Doing this via IMAP is required because simply deleting the files (like with rm) screws up cyrus` message spool. The message numbers are the only way I have to identify the messages that must be quarantined.[1] AFAICT, the 'order received' field Seamonkey displays gives you the correct "message number", i. e. the number from the file name used by cyrus. Hence when you use Seamonkey to delete a message, identified by the message number, it does not show up in the scanning report anymore. You can also verify this simply by looking at the message in Seamonkey and directly at the file that is stored in the spool directory. Messages with the same number are always identical seen both ways. IIRC, IMAP requires that message numbers, as seen by IMAP clients, must never change (because that could confuse the clients). So I don´t understand how the perl module can use different, apparently random, numbers as message numbers. Doing this seems very much broken by design and appears to be a bug that needs fixing. Its message number is *the* identifier, and a unique one per folder, for each message stored on an IMAP server. Expecting to get the message #1234 is what can be expected when you access message #1234. [1]: Of course, messages are being scanned before they are accepted by the MTA. With virus detection improving over time, there are always messages that were previously accepted and only later found to contain viruses. Such messages need to be automatically quarantined upon detection. Since I haven´t found a tool that does this, I´ve written one --- only to find out that it quarantines the wrong messages because the perl module uses random message numbers instead of the numbers the messages actually do have.