Bug #1959 for Mail-IMAPClient: bug: Mail::IMAPClient::BodyStructure::Parse chokes on some messages

Sun Jan 19 11:36:29 2003 adrian [...] quantumcat.org - Ticket created

From:	Adrian <adrian [...] quantumcat.org>
To:	bug-Mail-IMAPClient [...] rt.cpan.org
Subject:	bug: Mail::IMAPClient::BodyStructure::Parse chokes on some messages
Date:	Sun, 19 Jan 2003 11:34:31 -0500

Hello. I've found two bugs, actually: 1) In some circumstances, not all server output in response to the command FETCH BODYSTRUCTURE reaches Mail::IMAPClient::BodyStructure::Parser (see attached file bug1.txt) -- some of it is truncated. I suspect it's because sometimes the server splits the output on multiple lines (see raw_output.txt, which contains the server response to FETCH BODYSTRUCTURE in raw form), while IMAPClient expects all the relevant output in one line. 2) With the help of a really crude hack, I managed to get IMAPClient to pass all the FETCH BODYSTRUCTURE output to the BodyStructure parser, but the parser seems to choke on it (see attached bug2.txt). Environment details: Redhat Linux 8.0, kernel 2.4.19 Mail::IMAPClient 2.2.6 Perl: 5.8.0 UW imap 2001a How to reproduce the bug: as far as I can tell, both bugs only show up when processing messages containing MIME parts nested at least 3 levels deep. I just picked a random message that had a few pictures attached from my inbox and used the "Forward as attachment" function of my MUA to forward it to a test account. Calling FETCH BODYSTRUCTURE on such a message triggers the bugs. Files attached to this message: test.pl - the script I used to test the bugs bug1.txt - log of session that shows first bug in action bug2.txt - log of session that shows second bug raw_output.txt - raw output of FETCH BODYSTRUCTURE server response as returned by $imap->fetch(1, "BODYSTRUCTURE") to show that it is split across multiple lines For your convenience, I've also enabled tracing for Mail::IMAPClient::BodyStructure::Parser. Hope this helps. Adrian.

Message body is not shown because sender requested not to inline it.

Tue Jan 21 10:39:53 2003 DJKERNEN__NO_SOLICITING__ [...] cpan.org - Taken

Tue Jan 21 12:59:15 2003 DJKERNEN__NO_SOLICITING__ [...] cpan.org - Correspondence added

This is not really two bugs but one. The problem is caused by the body descpription field of the forwarded message subpart. This entire field is being passed as a literal string. The get_bodystructure method is parsing the output from the FETCH BODYSTRUCTURE IMAP client command in a naive manner, and is not expecting the output to be broken up the way it is. When the FETCH BODYSTRUCTURE output is obtained manually, the part that was delimited by a literal is placed inline without any indication that this chunk was once one discreet item. This is why the parse of the full bodystructure is failing. It is possible for me to fix this by carefully reassembling the entire response within the get_bodystructure method, replacing the "naive" code with something that can handle this style of response more gracefully. I'll update this ticket when that's ready. -- Dave K.

Wed Jan 22 10:20:39 2003 DJKERNEN__NO_SOLICITING__ [...] cpan.org - Status changed from 'new' to 'stalled'

Thu Jan 23 16:16:18 2003 DJKERNEN__NO_SOLICITING__ [...] cpan.org - Correspondence added

Enclosed is a patch. A similar bug exists in get_envelope, and this patch will not fix that bug. However both will be fixed in 2.2.7. Use the patch command to apply the patch. If you can't run patch (as will probably be the case if you are on NT) then let me know and I'll send a plain-text replacement for get_bodystructure. -- Dave K.

*** IMAPClient_pm.old Thu Jan 23 21:09:14 2003 --- IMAPClient.pm Thu Jan 23 21:10:26 2003 *************** *** 2009,2032 **** return undef; } sub get_bodystructure { my($self,$msg) = @_; unless ( eval {require Mail::IMAPClient::BodyStructure ; 1 } ) { $self->LastError("Unable to use get_bodystructure: $@\n"); return undef; } my $bs = ""; ! eval { $bs = Mail::IMAPClient::BodyStructure->new( ! grep( /bodystructure \(/i, # Wee! ;-) ! $self->fetch($msg,"BODYSTRUCTURE") ! ) ! )}; $self->_debug("get_bodystructure: msg $msg returns this ref: ". ( $bs ? " $bs" : " UNDEF" ) ."\n"); return $bs; } - sub get_envelope { my($self,$msg) = @_; unless ( eval {require Mail::IMAPClient::BodyStructure ; 1 } ) { --- 2009,2054 ---- return undef; } + # Updated to handle embedded literal strings sub get_bodystructure { my($self,$msg) = @_; unless ( eval {require Mail::IMAPClient::BodyStructure ; 1 } ) { $self->LastError("Unable to use get_bodystructure: $@\n"); return undef; } + my @out = $self->fetch($msg,"BODYSTRUCTURE"); my $bs = ""; ! my $output = grep( ! /BODYSTRUCTURE $/i, @out # Wee! ;-) ! ); ! if ( $output =~ /\r\n$/ ) { ! eval { $bs = Mail::IMAPClient::BodyStructure->new( $output )}; ! } else { ! $self->_debug("get_bodystructure: reassembling original response\n"); ! my $start = 0; ! foreach my $o (@{$self->{"History"}{$self->Transaction}}) { ! next unless $self->_is_output_or_literal($o); ! $self->_debug("o->[DATA] is ".$o->[DATA]."\n"); ! next unless $start or ! $o->[DATA] =~ /BODYSTRUCTURE \(/i and ++$start; # Hi, vi! ;-) ! if ( length($output) and $self->_is_literal($o) ) { ! my $data = $o->[DATA]; ! $data =~ s/"/\\"/g; ! $data =~ s/\(/\\\(/g; ! $data =~ s/$/\\\)/g; ! $output .= '"'.$data.'"'; ! } else { ! $output .= $o->[DATA] ; ! } ! $self->_debug("get_bodystructure: reassembled output=$output<END>\n"); ! } ! eval { $bs = Mail::IMAPClient::BodyStructure->new( $output )}; ! } $self->_debug("get_bodystructure: msg $msg returns this ref: ". ( $bs ? " $bs" : " UNDEF" ) ."\n"); return $bs; } sub get_envelope { my($self,$msg) = @_; unless ( eval {require Mail::IMAPClient::BodyStructure ; 1 } ) {

Thu Jan 23 16:16:19 2003 The RT System itself - Status changed from 'stalled' to 'open'

Thu Jan 23 16:18:09 2003 DJKERNEN__NO_SOLICITING__ [...] cpan.org - Status changed from 'open' to 'resolved'