Skip Menu |

This queue is for tickets about the Net-SFTP-Foreign CPAN distribution.

Report information
The Basics
Id: 102216
Status: resolved
Priority: 0/
Queue: Net-SFTP-Foreign

People
Owner: Nobody in particular
Requestors: REHSACK [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Wish: transparent solution for transfer directories and IO::All
Hi, thanks for Net::SFTP::Foreign \o/ In a customer project I encountered the situation that I'd like to handle the I/O layer of a process using IO::All, because the sources/targets reach from file over restful to sftp. So I'd start playing with IO::All::SFTP (based on LWP::Protocol::sftp based on Net::SFTP::Foreign). On key requirement when putting a file via sftp in the transfer directory situation is, that the file initially has to be written into a temporary location, flushed and finally renamed to the correct location. Given that you accept the challenge to enhance Net::SFTP::Foreign or LWP::Protocol::sftp (please feel free to move the ticket to another queue where it's appropriate) to support that wish - what would the best strategy? * add on_close callback/shell-code to Net::SFTP::Foreign * add write_to_tmp_location bool as session parameter * add explicit logic to LWP::Protocol::sftp for renaming from temporary directory (parameter!)? As far as I understood LWP::Protocol::sftp, the last one is the less favored idea ;) Thanks in advance for feedback. BTW: I'm happy to contribute a patch according you're decision how to solve this wish. Cheers, Jens
Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All
Date: Thu, 19 Feb 2015 16:35:15 +0000 (UTC)
To: "bug-Net-SFTP-Foreign [...] rt.cpan.org" <bug-Net-SFTP-Foreign [...] rt.cpan.org>
From: Salvador Fandino <sfandino [...] yahoo.com>
Show quoted text
----- Original Message -----
> From: Jens Rehsack via RT <bug-Net-SFTP-Foreign@rt.cpan.org> > To: > Cc: > Sent: Thursday, February 19, 2015 3:07 PM > Subject: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All > >T hu Feb 19 09:07:31 2015: Request 102216 was acted upon. > Transaction: Ticket created by REHSACK > Queue: Net-SFTP-Foreign > Subject: Wish: transparent solution for transfer directories and IO::All > Broken in: (no value) > Severity: (no value) > Owner: Nobody > Requestors: REHSACK@cpan.org > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > > > > Hi, > > thanks for Net::SFTP::Foreign \o/ > > In a customer project I encountered the situation that I'd like to handle > the I/O layer of a process using IO::All, because the sources/targets reach from > file over restful to sftp. So I'd start playing with IO::All::SFTP (based on > LWP::Protocol::sftp based on Net::SFTP::Foreign). > > On key requirement when putting a file via sftp in the transfer directory > situation is, that the file initially has to be written into a temporary > location, flushed and finally renamed to the correct location.
If I undestand your requirements correcly, that is what the "atomic" feature on the put method already does. You can also use just a "put" followed by a "rename" or "atomic_rename" operation.
> Given that you accept the challenge to enhance Net::SFTP::Foreign or > LWP::Protocol::sftp (please feel free to move the ticket to another queue where > it's appropriate) to support that wish - what would the best strategy? > > * add on_close callback/shell-code to Net::SFTP::Foreign > * add write_to_tmp_location bool as session parameter > * add explicit logic to LWP::Protocol::sftp for renaming from temporary > directory (parameter!)? > > As far as I understood LWP::Protocol::sftp, the last one is the less favored > idea ;)
Probably it was not a good idea to use LWP::Protocol::sftp as an intermediate layer between IO::All and Net::SFTP::Foreign in the first place!
Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All
Date: Thu, 19 Feb 2015 19:00:20 +0100
To: bug-Net-SFTP-Foreign [...] rt.cpan.org
From: Jens Rehsack <rehsack [...] cpan.org>
Show quoted text
> Am 19.02.2015 um 17:35 schrieb Salvador Fandiño via RT <bug-Net-SFTP-Foreign@rt.cpan.org>: > > <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > > > > > > > ----- Original Message -----
>> From: Jens Rehsack via RT <bug-Net-SFTP-Foreign@rt.cpan.org> >> To: >> Cc: >> Sent: Thursday, February 19, 2015 3:07 PM >> Subject: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All >> >> T hu Feb 19 09:07:31 2015: Request 102216 was acted upon. >> Transaction: Ticket created by REHSACK >> Queue: Net-SFTP-Foreign >> Subject: Wish: transparent solution for transfer directories and IO::All >> Broken in: (no value) >> Severity: (no value) >> Owner: Nobody >> Requestors: REHSACK@cpan.org >> Status: new >> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > >> >> >> Hi, >> >> thanks for Net::SFTP::Foreign \o/ >> >> In a customer project I encountered the situation that I'd like to handle >> the I/O layer of a process using IO::All, because the sources/targets reach from >> file over restful to sftp. So I'd start playing with IO::All::SFTP (based on >> LWP::Protocol::sftp based on Net::SFTP::Foreign). >> >> On key requirement when putting a file via sftp in the transfer directory >> situation is, that the file initially has to be written into a temporary >> location, flushed and finally renamed to the correct location.
> > > If I undestand your requirements correcly, that is what the "atomic" feature on the put method already does. > > You can also use just a "put" followed by a "rename" or "atomic_rename" operation.
I didn't realize that - yes, that would solve the issue. Maybe an atomic template would help to execute such a combination of operations through an IO::All operation. The benefit of IO::All (and also IO::All::SFTP) is the fact that AnyData can write encoded data directly to the target environment. Show quoted text
>> Given that you accept the challenge to enhance Net::SFTP::Foreign or >> LWP::Protocol::sftp (please feel free to move the ticket to another queue where >> it's appropriate) to support that wish - what would the best strategy? >> >> * add on_close callback/shell-code to Net::SFTP::Foreign >> * add write_to_tmp_location bool as session parameter >> * add explicit logic to LWP::Protocol::sftp for renaming from temporary >> directory (parameter!)? >> >> As far as I understood LWP::Protocol::sftp, the last one is the less favored >> idea ;)
> > > Probably it was not a good idea to use LWP::Protocol::sftp as an intermediate layer between IO::All and Net::SFTP::Foreign in the first place!
Do you have a better suggestion? Mind that IO::All has enormous benefits because of the transparency of the IO-implementation. Cheers -- Jens Rehsack rehsack@gmail.com
Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All
Date: Thu, 19 Feb 2015 19:01:33 +0000 (UTC)
To: "bug-Net-SFTP-Foreign [...] rt.cpan.org" <bug-Net-SFTP-Foreign [...] rt.cpan.org>
From: Salvador Fandino <sfandino [...] yahoo.com>
Show quoted text
----- Original Message -----
> From: Jens Rehsack via RT <bug-Net-SFTP-Foreign@rt.cpan.org> > To: > Cc: > Sent: Thursday, February 19, 2015 7:00 PM > Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All > > Queue: Net-SFTP-Foreign > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > > >
>> Am 19.02.2015 um 17:35 schrieb Salvador Fandiño via RT
> <bug-Net-SFTP-Foreign@rt.cpan.org>:
>> >> <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > >> >> >> >> >> >> ----- Original Message -----
>>> From: Jens Rehsack via RT <bug-Net-SFTP-Foreign@rt.cpan.org> >>> To: >>> Cc: >>> Sent: Thursday, February 19, 2015 3:07 PM >>> Subject: [rt.cpan.org #102216] Wish: transparent solution for transfer
> directories and IO::All
>>> >>> T hu Feb 19 09:07:31 2015: Request 102216 was acted upon. >>> Transaction: Ticket created by REHSACK >>> Queue: Net-SFTP-Foreign >>> Subject: Wish: transparent solution for transfer directories and
> IO::All
>>> Broken in: (no value) >>> Severity: (no value) >>> Owner: Nobody >>> Requestors: REHSACK@cpan.org >>> Status: new >>> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > >>> >>> >>> Hi, >>> >>> thanks for Net::SFTP::Foreign \o/ >>> >>> In a customer project I encountered the situation that I'd like to
> handle
>>> the I/O layer of a process using IO::All, because the sources/targets
> reach from
>>> file over restful to sftp. So I'd start playing with IO::All::SFTP
> (based on
>>> LWP::Protocol::sftp based on Net::SFTP::Foreign). >>> >>> On key requirement when putting a file via sftp in the transfer
> directory
>>> situation is, that the file initially has to be written into a
> temporary
>>> location, flushed and finally renamed to the correct location.
>> >> >> If I undestand your requirements correcly, that is what the
> "atomic" feature on the put method already does.
>> >> You can also use just a "put" followed by a "rename" or
> "atomic_rename" operation. > > I didn't realize that - yes, that would solve the issue. Maybe an atomic > template would help to execute such a combination of operations through an > IO::All operation. The benefit of IO::All (and also IO::All::SFTP) is the fact > that AnyData can write encoded data directly to the target environment. >
>>> Given that you accept the challenge to enhance Net::SFTP::Foreign or >>> LWP::Protocol::sftp (please feel free to move the ticket to another
> queue where
>>> it's appropriate) to support that wish - what would the best
> strategy?
>>> >>> * add on_close callback/shell-code to Net::SFTP::Foreign >>> * add write_to_tmp_location bool as session parameter >>> * add explicit logic to LWP::Protocol::sftp for renaming from temporary
>
>>> directory (parameter!)? >>> >>> As far as I understood LWP::Protocol::sftp, the last one is the less
> favored
>>> idea ;)
>> >> >> Probably it was not a good idea to use LWP::Protocol::sftp as an
> intermediate layer between IO::All and Net::SFTP::Foreign in the first place! > > Do you have a better suggestion? Mind that IO::All has enormous benefits because > of the transparency of the IO-implementation.
I don't know anything about IO:All internals or specifically how new backends can be plugged, but from the outside it seems to me that IO::All provides a feature rich API that more or less models a file system and the same is true for Net::SFTP::Foreign. On the other hand, LWP::Protocol::sftp provides a very narrow API that only supports a few basic operations (IIRC, get, put and ls), so it doesn't seem like a good idea to have it in the middle. I suppose the rigth way to support SFTP in IO::All would be to write a direct adapter for Net::SFTP::Foreign that could take advantage of most of its features.
Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All
Date: Thu, 19 Feb 2015 19:06:40 +0000 (UTC)
To: "bug-Net-SFTP-Foreign [...] rt.cpan.org" <bug-Net-SFTP-Foreign [...] rt.cpan.org>, "ingy [...] cpan.org" <ingy [...] cpan.org>
From: Salvador Fandino <sfandino [...] yahoo.com>
Hi Ingy, Could you give us some insight into the discussion here? https://rt.cpan.org/Ticket/Display.html?id=102216 Is there any documentation available about how to extend IO::All with new backends? Show quoted text
----- Original Message -----
> From: "Salvador \"Fandiño\" via RT" <bug-Net-SFTP-Foreign@rt.cpan.org> > To: > Cc: > Sent: Thursday, February 19, 2015 8:01 PM > Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All > > Queue: Net-SFTP-Foreign > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > > > > > > > > ----- Original Message -----
>> From: Jens Rehsack via RT <bug-Net-SFTP-Foreign@rt.cpan.org> >> To: >> Cc: >> Sent: Thursday, February 19, 2015 7:00 PM >> Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer
> directories and IO::All
>> >> Queue: Net-SFTP-Foreign >> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > >> >>
>>> Am 19.02.2015 um 17:35 schrieb Salvador Fandiño via RT
>> <bug-Net-SFTP-Foreign@rt.cpan.org>:
>>> >>> <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > >>> >>> >>> >>> >>> >>> ----- Original Message -----
>>>> From: Jens Rehsack via RT <bug-Net-SFTP-Foreign@rt.cpan.org> >>>> To: >>>> Cc: >>>> Sent: Thursday, February 19, 2015 3:07 PM >>>> Subject: [rt.cpan.org #102216] Wish: transparent solution for
> transfer
>> directories and IO::All
>>>> >>>> T hu Feb 19 09:07:31 2015: Request 102216 was acted upon. >>>> Transaction: Ticket created by REHSACK >>>> Queue: Net-SFTP-Foreign >>>> Subject: Wish: transparent solution for transfer directories
> and
>> IO::All
>>>> Broken in: (no value) >>>> Severity: (no value) >>>> Owner: Nobody >>>> Requestors: REHSACK@cpan.org >>>> Status: new >>>> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216
>>
>>>> >>>> >>>> Hi, >>>> >>>> thanks for Net::SFTP::Foreign \o/ >>>> >>>> In a customer project I encountered the situation that I'd
> like to
>> handle
>>>> the I/O layer of a process using IO::All, because the
> sources/targets
>> reach from
>>>> file over restful to sftp. So I'd start playing with
> IO::All::SFTP
>> (based on
>>>> LWP::Protocol::sftp based on Net::SFTP::Foreign). >>>> >>>> On key requirement when putting a file via sftp in the transfer
>> directory
>>>> situation is, that the file initially has to be written into a
>> temporary
>>>> location, flushed and finally renamed to the correct location.
>>> >>> >>> If I undestand your requirements correcly, that is what the
>> "atomic" feature on the put method already does.
>>> >>> You can also use just a "put" followed by a
> "rename" or
>> "atomic_rename" operation. >> >> I didn't realize that - yes, that would solve the issue. Maybe an
> atomic
>> template would help to execute such a combination of operations through an >> IO::All operation. The benefit of IO::All (and also IO::All::SFTP) is the
> fact
>> that AnyData can write encoded data directly to the target environment. >>
>>>> Given that you accept the challenge to enhance Net::SFTP::Foreign
> or
>>>> LWP::Protocol::sftp (please feel free to move the ticket to
> another
>> queue where
>>>> it's appropriate) to support that wish - what would the best
>> strategy?
>>>> >>>> * add on_close callback/shell-code to Net::SFTP::Foreign >>>> * add write_to_tmp_location bool as session parameter >>>> * add explicit logic to LWP::Protocol::sftp for renaming from
> temporary
>>
>>>> directory (parameter!)? >>>> >>>> As far as I understood LWP::Protocol::sftp, the last one is the
> less
>> favored
>>>> idea ;)
>>> >>> >>> Probably it was not a good idea to use LWP::Protocol::sftp as an
>> intermediate layer between IO::All and Net::SFTP::Foreign in the first
> place!
>> >> Do you have a better suggestion? Mind that IO::All has enormous benefits
> because
>> of the transparency of the IO-implementation.
> > > I don't know anything about IO:All internals or specifically how new > backends can be plugged, but from the outside it seems to me that IO::All > provides a feature rich API that more or less models a file system and the same > is true for Net::SFTP::Foreign. On the other hand, LWP::Protocol::sftp provides > a very narrow API that only supports a few basic operations (IIRC, get, put and > ls), so it doesn't seem like a good idea to have it in the middle. > > I suppose the rigth way to support SFTP in IO::All would be to write a direct > adapter for Net::SFTP::Foreign that could take advantage of most of its > features. >
Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All
Date: Thu, 19 Feb 2015 20:26:43 +0100
To: bug-Net-SFTP-Foreign [...] rt.cpan.org
From: Jens Rehsack <rehsack [...] cpan.org>
Show quoted text
> Am 19.02.2015 um 20:07 schrieb Salvador Fandiño via RT <bug-Net-SFTP-Foreign@rt.cpan.org>: > > <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > > > Hi Ingy, > > Could you give us some insight into the discussion here? > > https://rt.cpan.org/Ticket/Display.html?id=102216 > > Is there any documentation available about how to extend IO::All with new backends?
I think, the examples (DBM, LWP, FileSys) give me enough information, except Ingy want's to prevent specific deprecated API being used. Catch me in #toolchain or #dbi (I'm [Sno] on irc.perl.org). Cheers Show quoted text
> ----- Original Message -----
>> From: "Salvador \"Fandiño\" via RT" <bug-Net-SFTP-Foreign@rt.cpan.org> >> To: >> Cc: >> Sent: Thursday, February 19, 2015 8:01 PM >> Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer directories and IO::All >> >> Queue: Net-SFTP-Foreign >> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > >> >> >> >> >> >> >> ----- Original Message -----
>>> From: Jens Rehsack via RT <bug-Net-SFTP-Foreign@rt.cpan.org> >>> To: >>> Cc: >>> Sent: Thursday, February 19, 2015 7:00 PM >>> Subject: Re: [rt.cpan.org #102216] Wish: transparent solution for transfer
>> directories and IO::All
>>> >>> Queue: Net-SFTP-Foreign >>> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > >>> >>>
>>>> Am 19.02.2015 um 17:35 schrieb Salvador Fandiño via RT
>>> <bug-Net-SFTP-Foreign@rt.cpan.org>:
>>>> >>>> <URL: https://rt.cpan.org/Ticket/Display.html?id=102216 > >>>> >>>> >>>> >>>> >>>> >>>> ----- Original Message -----
>>>>> From: Jens Rehsack via RT <bug-Net-SFTP-Foreign@rt.cpan.org> >>>>> To: >>>>> Cc: >>>>> Sent: Thursday, February 19, 2015 3:07 PM >>>>> Subject: [rt.cpan.org #102216] Wish: transparent solution for
>> transfer
>>> directories and IO::All
>>>>> >>>>> T hu Feb 19 09:07:31 2015: Request 102216 was acted upon. >>>>> Transaction: Ticket created by REHSACK >>>>> Queue: Net-SFTP-Foreign >>>>> Subject: Wish: transparent solution for transfer directories
>> and
>>> IO::All
>>>>> Broken in: (no value) >>>>> Severity: (no value) >>>>> Owner: Nobody >>>>> Requestors: REHSACK@cpan.org >>>>> Status: new >>>>> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102216
>>>
>>>>> >>>>> >>>>> Hi, >>>>> >>>>> thanks for Net::SFTP::Foreign \o/ >>>>> >>>>> In a customer project I encountered the situation that I'd
>> like to
>>> handle
>>>>> the I/O layer of a process using IO::All, because the
>> sources/targets
>>> reach from
>>>>> file over restful to sftp. So I'd start playing with
>> IO::All::SFTP
>>> (based on
>>>>> LWP::Protocol::sftp based on Net::SFTP::Foreign). >>>>> >>>>> On key requirement when putting a file via sftp in the transfer
>>> directory
>>>>> situation is, that the file initially has to be written into a
>>> temporary
>>>>> location, flushed and finally renamed to the correct location.
>>>> >>>> >>>> If I undestand your requirements correcly, that is what the
>>> "atomic" feature on the put method already does.
>>>> >>>> You can also use just a "put" followed by a
>> "rename" or
>>> "atomic_rename" operation. >>> >>> I didn't realize that - yes, that would solve the issue. Maybe an
>> atomic
>>> template would help to execute such a combination of operations through an >>> IO::All operation. The benefit of IO::All (and also IO::All::SFTP) is the
>> fact
>>> that AnyData can write encoded data directly to the target environment. >>>
>>>>> Given that you accept the challenge to enhance Net::SFTP::Foreign
>> or
>>>>> LWP::Protocol::sftp (please feel free to move the ticket to
>> another
>>> queue where
>>>>> it's appropriate) to support that wish - what would the best
>>> strategy?
>>>>> >>>>> * add on_close callback/shell-code to Net::SFTP::Foreign >>>>> * add write_to_tmp_location bool as session parameter >>>>> * add explicit logic to LWP::Protocol::sftp for renaming from
>> temporary
>>>
>>>>> directory (parameter!)? >>>>> >>>>> As far as I understood LWP::Protocol::sftp, the last one is the
>> less
>>> favored
>>>>> idea ;)
>>>> >>>> >>>> Probably it was not a good idea to use LWP::Protocol::sftp as an
>>> intermediate layer between IO::All and Net::SFTP::Foreign in the first
>> place!
>>> >>> Do you have a better suggestion? Mind that IO::All has enormous benefits
>> because
>>> of the transparency of the IO-implementation.
>> >> >> I don't know anything about IO:All internals or specifically how new >> backends can be plugged, but from the outside it seems to me that IO::All >> provides a feature rich API that more or less models a file system and the same >> is true for Net::SFTP::Foreign. On the other hand, LWP::Protocol::sftp provides >> a very narrow API that only supports a few basic operations (IIRC, get, put and >> ls), so it doesn't seem like a good idea to have it in the middle. >> >> I suppose the rigth way to support SFTP in IO::All would be to write a direct >> adapter for Net::SFTP::Foreign that could take advantage of most of its >> features. >>
>
-- Jens Rehsack rehsack@gmail.com