Bug #123568 for Net-IMAP-Simple: SSL connections hang forever

Wed Nov 08 12:43:38 2017 cpanbugs2017 [...] wulf.eu.org - Ticket created

Subject:	SSL connections hang forever
Date:	Wed, 8 Nov 2017 18:39:37 +0100
To:	bug-Net-IMAP-Simple [...] rt.cpan.org
From:	Jörn Heissler <cpanbugs2017 [...] wulf.eu.org>

Hello, I'm using OTRS and it tries to fetch mails from an IMAPS (port 993) account. It uses Net:IMAP:Simple for this task. Sometimes the process hangs forever. From what I can see, the TLS socket is running in blocking mode and a read() system call hangs, while imap protocol messages are sent and received, i.e. the TLS connection itself was established already. I'm not entirely sure which module is at fault here, Net:IMAP:Simple or IO::Socket::SSL. My assumption is that Net:IMAP:Simple should use non-blocking sockets and handle timeouts itself. I don't have a full test case, but I found I can reproduce this by abusing iptables: iptables -A INPUT -s <imapserveraddress> -p tcp --sport 993 -m length 100:200 -j DROP Stack trace: /opt/otrs/bin/otrs.Daemon.pl:138 /opt/otrs/bin/otrs.Daemon.pl:316 /opt/otrs/bin/otrs.Daemon.pl:316 /opt/otrs/Kernel/System/Daemon/DaemonModules/SchedulerTaskWorker.pm:253 /opt/otrs/Kernel/System/Daemon/DaemonModules/SchedulerTaskWorker/Cron.pm:129 /opt/otrs/Kernel/System/Daemon/DaemonModules/SchedulerTaskWorker/Cron.pm:150 /opt/otrs/Kernel/System/Console/BaseCommand.pm:454 /opt/otrs/Kernel/System/Console/BaseCommand.pm:460 /opt/otrs/Kernel/System/Console/Command/Maint/PostMaster/MailAccountFetch.pm:94 /opt/otrs/Kernel/System/MailAccount.pm:440 /opt/otrs/Kernel/System/MailAccount/IMAP.pm:86 /opt/otrs/Kernel/System/MailAccount/IMAP.pm:257 /opt/otrs/Kernel/cpan-lib/Net/IMAP/Simple.pm:1021 /opt/otrs/Kernel/cpan-lib/Net/IMAP/Simple.pm:1210 while ( $res = $sock->getline ) { /usr/share/perl5/IO/Socket/SSL.pm:2043 sub getline OS: Ubuntu 16.04 Perl: 5.22.1 IO::Socket::SSL: 2.024 Net::IMAP::Simple: 1.2206 Best regards Joern Heissler

Thu Nov 16 07:55:39 2017 jettero [...] cpan.org - Correspondence added

If you DROP packets, the expected behavior is a hang. Try this with almost any network program and you'll see the exact same thing. Try it with netcat, openssh, mutt, socat, openssl, wget, curl... you name it. The alternative for the DROP case is a timeout where you set some arbitrary number (eg a minute, or 20 seconds or something) after which you _guess_ the connection is broken or something. Net::IMAP::SSL has such a timeout set, but IIRC it doesn't work somewhere in IO::Socket::SSL. In the not-DROP case, where something goes wrong, but the application doesn't receive RST, FIN, ACK (or whatever else); then the expected behavior is probably for the kernel itself to put the process in ready-wait (see loadavg) until something happens with the socket. TCP stacks can't just quit in the middle of a session, they have to do the RST or FIN thing. If they don't you end up with billions of TIME_WAIT sockets in netstat output. TIME_WAIT sockets should stay open literally forever or until they receive their FIN, but modern kernels usually time them out around a few hours (or some other arbitrary number). My point is, settings this to non-blocking and polling the connection probably won't fix a bug. It feels like a layer 2 or layer 3 problem to me. Fix the bugs on the network and this won't be an issue. You might be able to _work around_ the network bugs by making a non-blocking polling loop, but such a need is a matter of opinion. I say make a proxy for Net::IMAP::Simple or IO::Socket::SSL or whatever --- like you'd have to do with curl or ssh. -- If riding in an airplane is flying, then riding in a boat is swimming. 116 jumps, 48.6 minutes of freefall, 92.9 freefall miles.

Thu Nov 16 07:55:40 2017 The RT System itself - Status changed from 'new' to 'open'

Thu Nov 16 07:55:40 2017 jettero [...] cpan.org - Status changed from 'open' to 'rejected'

Thu Nov 16 09:22:54 2017 cpanbugs2017 [...] wulf.eu.org - Correspondence added

Subject:	Re: [rt.cpan.org #123568] SSL connections hang forever
Date:	Thu, 16 Nov 2017 15:22:38 +0100
To:	Paul Miller via RT <bug-Net-IMAP-Simple [...] rt.cpan.org>
From:	Jörn Heissler <cpanbugs2017 [...] wulf.eu.org>

Most software implements timeouts, for good reason. Some send keepalive mesages and abort if no response is received in a while. Some set the SO_KEEPALIVE socket option and the kernel will abort the connection. Others have an event loop with a timer and close the connection. Some arm an alarm() or start a timeout thread or whatever. And some software doesn't implement any timeout at all, and then the program will hang forever. * Netcat has -w. * Openssh sends keepalive messages on application level and uses SO_KEEPALIVE. * Mutt is lacking a timeout which can be annoying, but reacts to ctrl-c too. * socat has "keepalive" to enable SO_KEEPALIVE. * wget has --timeout. * curl has --max-time and at least in the lib some more sophisticated options. So 1 out of your 6 doesn't implement timeouts. It's normal behaviour of networks and the internet to occasionally drop packets. Usually it doesn't matter, but sometimes one end closes the connection and the other end considers the connection to be still open. This CANNOT be fixed on the network, software has to cope with this and implement some sort of timeout.

Download signature.asc
application/pgp-signature 833b

Message body not shown because it is not plain text.

Thu Nov 16 10:45:53 2017 jettero [...] cpan.org - Correspondence added

Yes, I agree. I said that. There's a 90 second timeout in the package. You can tune it to your liking. I was just trying to cover a lot of bases in case I misunderstood the problem. Non-blocking mode is not the answer either way though. That's for cases where you expect no answer, rather than cases where you might not get one if something goes wrong. Also, if the timeout's not working, it's probably the layer above. -- If riding in an airplane is flying, then riding in a boat is swimming. 116 jumps, 48.6 minutes of freefall, 92.9 freefall miles.