Bug #110644 for IO-Compress: Tutorial about writing a new IO::Compress module

Sun Dec 27 04:22:15 2015 marius [...] ieval.ro - Ticket created

Subject:

Tutorial about writing a new IO::Compress module

Hello, I'd like to write an IO::Compress module. However, there seems to be no documentation about the steps that should be followed. I figure I should subclass IO::Compress::Base and IO::Uncompress::Base, but neither of the modules have documentation about what methods must/should be implemented. I guess I can find this out by reading the source, but it would be much better if a tutorial or some documentation would be available.

Thu Dec 31 06:47:34 2015 pmqs [...] cpan.org - Correspondence added

On Sun Dec 27 04:22:15 2015, MGV wrote: Show quoted text

> Hello, > > I'd like to write an IO::Compress module. However, there seems to be > no documentation about the steps that should be followed. I figure I > should subclass IO::Compress::Base and IO::Uncompress::Base, but > neither of the modules have documentation about what methods > must/should be implemented. I guess I can find this out by reading the > source, but it would be much better if a tutorial or some > documentation would be available.

Hey Marius, writing a tutorial for this would be difficult - the code isn't well enough structured to allow an easy tutorial. The source is probably your best bet. Also - not all compression libraries are suitable for the IO:Compress treatment. Which were you planning to implement? Please shout if you get stuck & need a hand. sorry for the delay in replying. Christmas got in the way. Paul

Thu Dec 31 06:47:34 2015 The RT System itself - Status changed from 'new' to 'open'

Thu Dec 31 07:01:34 2015 marius [...] ieval.ro - Correspondence added

Subject:	Re: [rt.cpan.org #110644] Tutorial about writing a new IO::Compress module
Date:	Thu, 31 Dec 2015 14:00:31 +0200
To:	"Paul Marquess via RT" <bug-IO-Compress [...] rt.cpan.org>
From:	Marius Gavrilescu <marius [...] ieval.ro>

"Paul Marquess via RT" <bug-IO-Compress@rt.cpan.org> writes: Show quoted text

> writing a tutorial for this would be difficult - the code isn't well > enough structured to allow an easy tutorial. The source is probably > your best bet. Also - not all compression libraries are suitable for > the IO:Compress treatment. Which were you planning to implement?

When I said "tutorial" I was thinking of something of the form "Here's a list of methods you must/should implement and what each of them should do". I am implementing a module for the Brotli compression format, which is the format originally used in web fonts and more recently as a HTTP Content-Encoding, thus replacing gzip and deflate. At the moment it's only implemented in Firefox, but moreimplementations are coming. The official repository is https://github.com/google/brotli/ I will release an initial version today. It will only handle decompression (I still have to figure out how to use XS with C++, as the official compression code is C++ while the decompression code is C), and it will be a simple wrapper around the official API. In the following weeks, after I read the source of IO::{Unc,C}ompress::Base, I'll release updated versions that support compression and follow the IO::{Unc,C}ompress API. Show quoted text

> Please shout if you get stuck & need a hand.

Alright. Show quoted text

> sorry for the delay in replying. Christmas got in the way.

No problem, thanks for your reply! -- Marius Gavrilescu

Download signature.asc
application/pgp-signature 818b

Message body not shown because it is not plain text.

Thu Dec 31 12:59:21 2015 pmqs [...] cpan.org - Correspondence added

On Thu Dec 31 07:01:34 2015, MGV wrote: Show quoted text

> "Paul Marquess via RT" <bug-IO-Compress@rt.cpan.org> writes: >

> > writing a tutorial for this would be difficult - the code isn't well > > enough structured to allow an easy tutorial. The source is probably > > your best bet. Also - not all compression libraries are suitable for > > the IO:Compress treatment. Which were you planning to implement?

> > When I said "tutorial" I was thinking of something of the form "Here's a > list of methods you must/should implement and what each of them should do".

That would be good to do. Will add it to my (long) list of things to deal with. Show quoted text

> I am implementing a module for the Brotli compression format, which is > the format originally used in web fonts and more recently as a HTTP > Content-Encoding, thus replacing gzip and deflate. At the moment it's > only implemented in Firefox, but moreimplementations are coming. > > The official repository is https://github.com/google/brotli/

Yep - looked at that one a while ago. It's on my list of compression formats to keep an eye on. My main problem is getting the time to devote to my perl modules at the moment. If you want this to work with IO::Compress it has to use the Brolti streaming interface that allows compression/uncompression to be done a bit at a time. I *think* Brolti also has an interface that takes the complete compressed input and uncompresses (and the same for compression) -- that won't work with IO::Compress. Show quoted text

> I will release an initial version today. It will only handle > decompression (I still have to figure out how to use XS with C++, as the > official compression code is C++ while the decompression code is C), and > it will be a simple wrapper around the official API.

That makes sense. The IO::Compress code assumes there is a module that handles the API to the compresion library. For example all the zlib based IO::Compress modules (gzip, deflate, zip) use Compress::Raw::Zlib to access the library. Show quoted text

> In the following > weeks, after I read the source of IO::{Unc,C}ompress::Base, I'll release > updated versions that support compression and follow the > IO::{Unc,C}ompress API. >

> > Please shout if you get stuck & need a hand.

> > Alright. >

> > sorry for the delay in replying. Christmas got in the way.

> > No problem, thanks for your reply!

cheers Paul ps - I see Perl6 has a Brolti module. See https://github.com/sylvarant/Compress-Brotli

Thu Dec 31 14:15:06 2015 marius [...] ieval.ro - Correspondence added

Subject:	Re: [rt.cpan.org #110644] Tutorial about writing a new IO::Compress module
Date:	Thu, 31 Dec 2015 21:14:16 +0200
To:	"Paul Marquess via RT" <bug-IO-Compress [...] rt.cpan.org>
From:	Marius Gavrilescu <marius [...] ieval.ro>

"Paul Marquess via RT" <bug-IO-Compress@rt.cpan.org> writes: Show quoted text

>> I will release an initial version today. It will only handle >> decompression (I still have to figure out how to use XS with C++, as the >> official compression code is C++ while the decompression code is C), and >> it will be a simple wrapper around the official API.

> > That makes sense. The IO::Compress code assumes there is a module that > handles the API to the compresion library. For example all the zlib > based IO::Compress modules (gzip, deflate, zip) use > Compress::Raw::Zlib to access the library.

And IO-Compress-Brotli-0.001 (which can't compress yet) is now on the CPAN: https://metacpan.org/release/MGV/IO-Compress-Brotli-0.001 It wraps both the one-shot (whole-buffer) and the streaming interfaces of the Brotli decompression library. It also passes the official test suite (with the streaming interface, as the one-shot interface can't reasonably handle 3 of the test cases). I'll read up on Compress::Raw and IO::{Unc,C}ompress::Base in the following weeks and try to implement conforming to their APIs. Show quoted text

> ps - I see Perl6 has a Brolti module. See https://github.com/sylvarant/Compress-Brotli

Ooh, that's nice. But it does not seem to offer a streaming interface... Also, I didn't know libbrotli existed! I just shipped the relevant part of the Brotli source code and linked against it. Happy new year! -- Marius Gavrilescu

Download signature.asc
application/pgp-signature 818b

Message body not shown because it is not plain text.

Fri Jan 01 04:45:05 2016 pmqs [...] cpan.org - Correspondence added

... Show quoted text

> > And IO-Compress-Brotli-0.001 (which can't compress yet) is now on the > CPAN: > https://metacpan.org/release/MGV/IO-Compress-Brotli-0.001

Excellent! Show quoted text

> It wraps both the one-shot (whole-buffer) and the streaming interfaces > of the Brotli decompression library. It also passes the official test > suite (with the streaming interface, as the one-shot interface can't > reasonably handle 3 of the test cases). > > I'll read up on Compress::Raw and IO::{Unc,C}ompress::Base in the > following weeks and try to implement conforming to their APIs.

OK. Drop me a line if you get stuck on anything. Paul