Bug #85710 for Dist-Zilla-Plugin-ReadmeAnyFromPod: Writing to root fails for type 'pod' if content contains wide characters

Wed May 29 14:11:45 2013 dagolden [...] cpan.org - Ticket created

Subject:

Writing to root fails for type 'pod' if content contains wide characters

My plugin bundle does this: ReadmeAnyFromPod => ReadmeInRoot => { type => 'pod', filename => 'README.pod', location => 'root', } If my POD contains wide characters, the File::Slurp dies like this: [@Filter/ReadmeInRoot] Override README.pod in root Wide character in syswrite at /Users/david/.perlbrew /libs/19.0@std/lib/perl5/File/Slurp.pm line 506. Could you please see about detecting the "=encoding" line or something for the 'pod' type? Thank you, David

Wed May 29 18:04:26 2013 RTHOMPSON [...] cpan.org - Correspondence added

On Wed May 29 14:11:45 2013, DAGOLDEN wrote: Show quoted text

> My plugin bundle does this: > > ReadmeAnyFromPod => ReadmeInRoot => { > type => 'pod', > filename => 'README.pod', > location => 'root', > } > > If my POD contains wide characters, the File::Slurp dies like this: > > [@Filter/ReadmeInRoot] Override README.pod in root > Wide character in syswrite at /Users/david/.perlbrew > /libs/19.0@std/lib/perl5/File/Slurp.pm line 506. > > Could you please see about detecting the "=encoding" line or something > for the 'pod' type? > > Thank you, > David

Thanks for reporting this. It looks like File::Slurp more or less doesn't support character encodings. I have to admit I know almost nothing about non-ASCII text handling, so I don't really know how to fix this issue. If you want to point me in the right direction, I can try to figure it out myself, or you could submit a pull request or send me a patch if you know how to do it and have the time. I would certainly want to merge a fix.

Wed May 29 18:04:26 2013 The RT System itself - Status changed from 'new' to 'open'

Wed May 29 20:37:32 2013 dagolden [...] cpan.org - Correspondence added

File::Slurp is half the problem. (It's really bad, actually.) The other problem is that Dist::Zilla doesn't deal with encodings -- or rather it deals with content as raw bytes. I'm not sure there's a general solution, but I'll poke at it and send a pull request.

Wed May 29 20:43:38 2013 RTHOMPSON [...] cpan.org - Correspondence added

On Wed May 29 20:37:32 2013, DAGOLDEN wrote: Show quoted text

> File::Slurp is half the problem. (It's really bad, actually.)

Yes, I realized that pretty quickly. I have no issues with abandoning it. Show quoted text

> The other problem is that Dist::Zilla doesn't deal with encodings -- > or rather it deals with content as raw bytes. > > I'm not sure there's a general solution, but I'll poke at it and send > a pull request.

If you need to add another option to the plugin so the user can manually specify the encoding, that's fine, although obviously less optimal than magically detecting the encoding.

Wed May 29 21:01:29 2013 dagolden [...] cpan.org - Correspondence added

What I've discovered is that I can't replicate it from a test corpus. That makes me suspect that there's an interaction with Pod::Weaver and the Dist::Zilla plugin for it. Yuck. If I can't quickly replicate it, I might try something quick and dirty to try to fix it and test it manually on the dist that had the problem. Not how I like to work, but I don't want to burden your test suite with all of Pod::Weaver, either.

Wed May 29 21:52:53 2013 RTHOMPSON [...] cpan.org - Correspondence added

On Wed May 29 21:01:29 2013, DAGOLDEN wrote: Show quoted text

> What I've discovered is that I can't replicate it from a test corpus. > That makes me suspect that there's an interaction with Pod::Weaver and > the Dist::Zilla plugin for it. > > Yuck. If I can't quickly replicate it, I might try something quick > and dirty to try to fix it and test it manually on the dist that had > the problem. > > Not how I like to work, but I don't want to burden your test suite > with all of Pod::Weaver, either.

I don't necessarily mind having Pod::Weaver in my test suite. I use Pod::Weaver anyway, so I have it installed. On the other hand, it could be wasteful burdening all the smoke testing servers with the need to install Pod::Weaver every time they test my dist. Anyway, if you figure out the cause it it requires Pod::Weaver, I don't personally have a problem with adding it as a dependency. (Maybe it could be an optional dependency, and this test would be skipped if it's not already installed?)

Wed May 29 22:02:05 2013 dagolden [...] cpan.org - Correspondence added

I talked to Ricardo and it's a tricky problem. Dist::Zilla reads files raw, so gets UTF-8 encoded octets. Then Pod::Weaver might do something with (wide) characters. So you wind up with a string mixed between octets and characters -- which isn't something you can then encode or you'd double-encode the original UTF-8 octets (I think). So, I'm going to punt on this. Feel free to close the ticket.

Thu May 30 00:48:04 2013 RTHOMPSON [...] cpan.org - Correspondence added

On Wed May 29 22:02:05 2013, DAGOLDEN wrote: Show quoted text

> I talked to Ricardo and it's a tricky problem. > > Dist::Zilla reads files raw, so gets UTF-8 encoded octets. Then > Pod::Weaver might do something with (wide) characters. So you wind > up with a string mixed between octets and characters -- which isn't > something you can then encode or you'd double-encode the original > UTF-8 octets (I think). > > So, I'm going to punt on this. Feel free to close the ticket.

Ok, feel free to open a new ticket if and when Dist::Zilla and Pod::Weaver learn to play nice. I'll be glad to merge a fix. Thanks for working on this!

Thu May 30 00:48:04 2013 RTHOMPSON [...] cpan.org - Status changed from 'open' to 'rejected'