Bug #132768 for Net-SFTP-Foreign: directory listing failed: Couldn't read directory '/foo': bad packet sequence, expected 3, got 4

Thu Jun 04 06:50:57 2020 rosert.20.ftumsh [...] xoxy.net - Ticket created

Subject:	directory listing failed: Couldn't read directory '/foo': bad packet sequence, expected 3, got 4
Date:	Thu, 4 Jun 2020 11:27:15 +0100
To:	bug-net-sftp-foreign [...] rt.cpan.org
From:	rosert.20.ftumsh [...] xoxy.net

Hello, It looks like the 1.90_1 doesn't work. I get the error ' directory listing failed: Couldn't read directory '/foo': bad packet sequence, expected 3, got 4' when the directory is empty and also when it has a certain number of files. That number varies depending upon the size of file. For example 14 or more files of 1 byte in size cause the error. 13 or fewer works as expected. When I changed to 1000byte size files I still got the error with 13 files. (I didn't test fewer files). # This is Net::SFTP::Foreign 1.90_01 # Loaded from /usr/local/share/perl5/Net/SFTP/Foreign.pm # Running on Perl v5.10.1 for linux # debug set to 65536 # ~0 is 18446744073709551615 # Using backend Net::SFTP::Foreign::Backend::Unix 1.88_02 OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013 Warmly John

Thu Jun 04 07:04:10 2020 salva [...] cpan.org - Correspondence added

On Thu Jun 04 06:50:57 2020, rosert.20.ftumsh@xoxy.net wrote: Show quoted text

> Hello,

Hi! Show quoted text

> It looks like the 1.90_1 doesn't work. > > I get the error ' directory listing failed: Couldn't read directory > '/foo': bad packet sequence, expected 3, got 4' >

Could you enable debugging, rerun your program and post here (or send it directly to me by email, if you thing it may include any sensible information)? Just add the following line at the beginning of your script: $Net::SFTP::Foreign::debug = -1;

Thu Jun 04 07:04:10 2020 The RT System itself - Status changed from 'new' to 'open'

Thu Jun 04 08:08:33 2020 salva [...] cpan.org - Correspondence added

On Thu Jun 04 06:50:57 2020, rosert.20.ftumsh@xoxy.net wrote: Show quoted text

> Hello, > > It looks like the 1.90_1 doesn't work.

Also, 1.90_01 is working for me, and by the error you get it could actually be an interoperability issue. So, could you tell me which software and version is running on the remote side? You can get it connecting with netcat or some similar tool to the remote server. For instance: $ nc localhost 22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4 ^C

Thu Jun 04 09:02:17 2020 rosert.20.ftumsh [...] xoxy.net - Correspondence added

Subject:	Re: [rt.cpan.org #132768] directory listing failed: Couldn't read directory '/foo': bad packet sequence, expected 3, got 4
Date:	Thu, 4 Jun 2020 14:01:59 +0100
To:	bug-net-sftp-foreign [...] rt.cpan.org <bug-net-sftp-foreign [...] rt.cpan.org>
From:	rosert.20.ftumsh [...] xoxy.net

Hello! Yes, I'll send what I can. COuld you keep it off the public page please? Warmly John On 04/06/2020 13:08, Salvador Fandino Garcia via RT - bug-Net-SFTP-Foreign@rt.cpan.org wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=132768 > > > On Thu Jun 04 06:50:57 2020, rosert.20.ftumsh@xoxy.net wrote:

>> Hello, >> >> It looks like the 1.90_1 doesn't work.

> Also, 1.90_01 is working for me, and by the error you get it could actually be an interoperability issue. > > So, could you tell me which software and version is running on the remote side? > > You can get it connecting with netcat or some similar tool to the remote server. For instance: > > $ nc localhost 22 > SSH-2.0-OpenSSH_8.2p1 Ubuntu-4 > ^C > > >

Thu Jun 04 09:05:09 2020 rosert.20.ftumsh [...] xoxy.net - Correspondence added

Subject:	Re: [rt.cpan.org #132768] directory listing failed: Couldn't read directory '/foo': bad packet sequence, expected 3, got 4
Date:	Thu, 4 Jun 2020 14:04:47 +0100
To:	bug-net-sftp-foreign [...] rt.cpan.org <bug-net-sftp-foreign [...] rt.cpan.org>
From:	rosert.20.ftumsh [...] xoxy.net

As requested. On 04/06/2020 13:08, Salvador Fandino Garcia via RT - bug-Net-SFTP-Foreign@rt.cpan.org wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=132768 > > > On Thu Jun 04 06:50:57 2020, rosert.20.ftumsh@xoxy.net wrote:

>> Hello, >> >> It looks like the 1.90_1 doesn't work.

> Also, 1.90_01 is working for me, and by the error you get it could actually be an interoperability issue. > > So, could you tell me which software and version is running on the remote side? > > You can get it connecting with netcat or some similar tool to the remote server. For instance: > > $ nc localhost 22 > SSH-2.0-OpenSSH_8.2p1 Ubuntu-4 > ^C > > >

Message body is not shown because sender requested not to inline it.

Thu Jun 04 09:22:52 2020 rosert.20.ftumsh [...] xoxy.net - Correspondence added

Subject:	Re: [rt.cpan.org #132768] directory listing failed: Couldn't read directory '/foo': bad packet sequence, expected 3, got 4
Date:	Thu, 4 Jun 2020 14:22:28 +0100
To:	bug-net-sftp-foreign [...] rt.cpan.org <bug-net-sftp-foreign [...] rt.cpan.org>
From:	rosert.20.ftumsh [...] xoxy.net

Just to be clear, this works with every other connection we have and they are numerous. Here's code, cut down a little... my%opts= ( host => $args->{server_name}, port => $args->{port_} || 22, user => $args->{user_}, more => [ qw(-oStrictHostKeyChecking=no -oLogLevel=ERROR)] ); push@{$opts{more}}, '-vvv'if$args->{debug}; $Net::SFTP::Foreign::debug= $args->{debug} if$args->{debug}; if( $args->{private_key} ) { $opts{key_path} = $key_file; $opts{passphrase} = $args->{password_} if$args->{password_}; } elsif( $args->{password_} ) { $opts{password} = $args->{password_}; } $sftp= Net::SFTP::Foreign->new(%opts); $sftp->setcwd($args->{dest_dir}) or die"unable to change cwd: ". $sftp->error; my$file_list= $sftp->ls or die'directory listing failed: '. $sftp->error; On 04/06/2020 13:08, Salvador Fandino Garcia via RT - bug-Net-SFTP-Foreign@rt.cpan.org wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=132768 > > > On Thu Jun 04 06:50:57 2020, rosert.20.ftumsh@xoxy.net wrote:

>> Hello, >> >> It looks like the 1.90_1 doesn't work.

> Also, 1.90_01 is working for me, and by the error you get it could actually be an interoperability issue. > > So, could you tell me which software and version is running on the remote side? > > You can get it connecting with netcat or some similar tool to the remote server. For instance: > > $ nc localhost 22 > SSH-2.0-OpenSSH_8.2p1 Ubuntu-4 > ^C > > >

Thu Jun 04 10:40:43 2020 sfandino [...] yahoo.com - Correspondence added

Subject:	Re: [rt.cpan.org #132768] directory listing failed: Couldn't read directory '/foo': bad packet sequence, expected 3, got 4
Date:	Thu, 4 Jun 2020 16:40:24 +0200
To:	bug-Net-SFTP-Foreign [...] rt.cpan.org
From:	Salvador Fandiño <sfandino [...] yahoo.com>

On 4/6/20 15:05, rosert.20.ftumsh@xoxy.net via RT wrote: Show quoted text

> Queue: Net-SFTP-Foreign > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=132768 > > > As requested.

You have set $Net::SFTP::Foreign::debug to 65536 but I need you to set it to -1 in order to enable all the debugging flags. Show quoted text

> On 04/06/2020 13:08, Salvador Fandino Garcia via RT - > bug-Net-SFTP-Foreign@rt.cpan.org wrote:

>> <URL: https://rt.cpan.org/Ticket/Display.html?id=132768 > >> >> On Thu Jun 04 06:50:57 2020, rosert.20.ftumsh@xoxy.net wrote:

>>> Hello, >>> >>> It looks like the 1.90_1 doesn't work.

>> Also, 1.90_01 is working for me, and by the error you get it could actually be an interoperability issue. >> >> So, could you tell me which software and version is running on the remote side? >> >> You can get it connecting with netcat or some similar tool to the remote server. For instance: >> >> $ nc localhost 22 >> SSH-2.0-OpenSSH_8.2p1 Ubuntu-4 >> ^C >> >> >>

Thu Jun 04 10:47:40 2020 sfandino [...] yahoo.com - Correspondence added

Subject:	Re: [rt.cpan.org #132768] directory listing failed: Couldn't read directory '/foo': bad packet sequence, expected 3, got 4
Date:	Thu, 4 Jun 2020 16:47:30 +0200
To:	bug-Net-SFTP-Foreign [...] rt.cpan.org
From:	Salvador Fandiño <sfandino [...] yahoo.com>

On 4/6/20 15:22, rosert.20.ftumsh@xoxy.net via RT wrote: Show quoted text

> Queue: Net-SFTP-Foreign > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=132768 > > > Just to be clear, this works with every other connection we have and > they are numerous.

Yeah, that looks like an interoperability issue. Probably AWS is using their own implementation of the SFTP protocol. In any case: 1) Are you able to use any previous version of Net::SFTP::Foreign to connect to that AWS service and list or transfer files? 2) Are you able to connect to other servers using Net::SFTP::Foreign version 1.90_01?

Wed Jun 24 08:48:11 2020 salva [...] cpan.org - Correspondence added

On Thu Jun 04 10:47:40 2020, sfandino@yahoo.com wrote: Show quoted text

> > On 4/6/20 15:22, rosert.20.ftumsh@xoxy.net via RT wrote:

> > Queue: Net-SFTP-Foreign > > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=132768 > > > > > Just to be clear, this works with every other connection we have and > > they are numerous.

> > Yeah, that looks like an interoperability issue. Probably AWS is using > their own implementation of the SFTP protocol.

Yeah, it seems that AWS_SFTP 1.0 server can reply in an out of order fashion. In theory, that is something that clients should support, but in practice, and until now, no server was doing it and Net::SFTP::Foreign relies on reply packets arriving in order. As a work around you can disable pipelining: $sftp = Net::SFPT::Foreign->new(..., queue_size => 1) Unfortunatelly, disabling pipelining would probably reduce the transfer speed a lot.

Wed Jun 24 14:54:56 2020 salva [...] cpan.org - Correspondence added

On Wed Jun 24 08:48:11 2020, SALVA wrote: Show quoted text

> On Thu Jun 04 10:47:40 2020, sfandino@yahoo.com wrote:

> > > > On 4/6/20 15:22, rosert.20.ftumsh@xoxy.net via RT wrote:

> > > Queue: Net-SFTP-Foreign > > > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=132768 > > > > > > > Just to be clear, this works with every other connection we have > > > and > > > they are numerous.

> > > > Yeah, that looks like an interoperability issue. Probably AWS is > > using > > their own implementation of the SFTP protocol.

> > Yeah, it seems that AWS_SFTP 1.0 server can reply in an out of order > fashion. In theory, that is something that clients should support, but > in practice, and until now, no server was doing it and > Net::SFTP::Foreign relies on reply packets arriving in order. > > As a work around you can disable pipelining: > > $sftp = Net::SFPT::Foreign->new(..., queue_size => 1) > > > Unfortunatelly, disabling pipelining would probably reduce the > transfer speed a lot.

I have released two new versions: the stable 1.91 that fixes the ls queue size bug and a development version, 1.92_01, that adds support for incoming out of order responses. Could you check if 1.92_01 works for you? https://metacpan.org/release/SALVA/Net-SFTP-Foreign-1.92_01