Bug #91530 for Module-ScanDeps: Missing MIME\types.db from pp-generated code

Thu Dec 19 19:46:55 2013 md5159 [...] mclink.it - Ticket created

Subject:	Missing MIME\types.db from pp-generated code
Date:	Fri, 20 Dec 2013 01:46:41 +0100
To:	bug-PAR-Packer [...] rt.cpan.org
From:	ric <md5159 [...] mclink.it>

I've an issue, for a missing of MIME:Types.db from code generated via pp of PAR::Packer. Development environment is: XP SP3 machine within VirtualBox Strawberry Perl 5.18.1.1 PAR::Packer 1.017 MIME:Lite 3.030 Interpreted code (i.e. code invoked via "perl -w scriptname.pl") runs fine within this environment, so I would exclude issues on the script. pp compilation is invoked in a very plain way as pp -v -o scriptname.exe scriptname.pl and does not output any warning. When the executable gets transferred on a "real" environment (an independent, XP SP3 "true" box), program fails with a "cannot open type database in MIME\types.db: No such file or directory" As a matter of fact, if I enter the cache folder within TEMP (where pp-compiled scripts expand their contents), and navigate downwards to \inc\lib\MIME, I find 4 modules there (base64.pm, lite.pm, type.pm, types.pm), but no types.db whatsoever. I've also tried to include explicitly MIME:Lite via -M pp -M MIME:Lite -v -o scriptname.exe scriptname.pl and also, specifying source and destination of the file to be included, pp -a "c:\strawberry\perl\site\lib\MIME\types.db;/inc/lib/MIME/types.db" -v -o scriptname.exe scriptname.pl still, to no extent. In the last form, I've also opened the zipped executable, and requested file is not where it should be. Can't say whether this is an issue or a pitfall on my side; whatever the case, any suggestion/workaround is greatly appreciated. regards ric

Fri Dec 20 03:24:14 2013 RSCHUPP [...] cpan.org - Queue changed from PAR-Packer to Module-ScanDeps

Fri Dec 20 03:24:58 2013 RSCHUPP [...] cpan.org - Status changed from 'new' to 'open'

Fri Dec 20 03:24:59 2013 RSCHUPP [...] cpan.org - Broken in 1.12 added

Fri Dec 20 03:25:36 2013 RSCHUPP [...] cpan.org - Taken

Sat Dec 21 06:52:52 2013 RSCHUPP [...] cpan.org - Correspondence added

On 2013-12-19 19:46:55, md5159@mclink.it wrote: Show quoted text

> > I've an issue, for a missing of MIME:Types.db from code generated via pp > of PAR::Packer. > ... > Can't say whether this is an issue or a pitfall on my side; whatever the > case, any suggestion/workaround is greatly appreciated.

Definitely an issue, but it's complicated. If you don't want to read through my ramblings below, skip to the bottom for a workaround :) When a pp-packed executable is missing a module or data file, this can usually be worked around by using pp options -M (for a missing module) or -a (for a plain file), though to get the latter "right" may need some experimentation (or insight how some module searches for its data file). BTW, even if your workaround works, you might submit a bug report for Module::ScanDeps (which is used by pp to scan your script for all modules it uses). I can usually make the workaround unnecessery by adding another explicit rule to Module::ScanDeps. In your case, MIME::Types (it's not in MIME::Lite, but used by MIME::Lite under the hood) uses the data file MIME/types.db. Your idea to use "pp -a ..." to pack this file explicitly is correct, but can't work because MIME::Types does the following to open this file: open FH, "<:encoding(utf8)", File::Spec->catfile(dirname(__FILE__), "types.db") This assumes that (when perl parses MIME/Types.pm) __FILE__ is set to the absolute pathname where MIME/Types.pm was found. This is probably true for 99% of all use cases, but it always breaks in a packed executable. The reason is that any module is transformed ("filtered") when it is packed. The default filter (PodStrip) strips POD from the module. To compensate for that (so that a die() or warn() from inside the module shows the "correct" line number) it adds "# line N ..." statements. Especially it adds # line 1 "MIME/Types.pm" at the start of the packed MIME/Types.pm. As a side effect this sets __FILE__ to "MIME/Types.pm". So we must use the following options for pp: -a '/path/to/your/MIME/types.db;lib/MIME/types.db' to pack MIME/types.db as a "sibling" of MIME/Types.pm AND suppress the application of the PodStrip filter for MIME/Types.pm: -F PatchContent=MIME/Types.pm (or set PAR_VERBATIM=1 in the environment when packing to generally suppress filtering). But wait, there's more :( With the above you will still get cannot open type database in /tmp/par-726f646572696368/cache-5be06ead4dff6a6e16c2df7d03b59c0dbf5a8c02/inc/lib/MIME/types.db: No such file or directory even though this file exists! The reason is the argument "<:encoding(utf8)" to open(). It implicitly does "require PerlIO::encoding". Since this module hasn't been packed, open() fails even before actually trying to open the file. Actually, this is a bug in Module::ScanDeps, it's supposed to look out for for patterns like this and infer a dependency on PerlIO and PerlIO::encoding. I've fixed this in Module::ScanDeps, but until its next release the workaround is to use the pp option -M PerlIO::encoding So the whole workaround is pp -a "/path/to/your/MIME/types.db;lib/MIME/types.db" \ -F PatchContent=MIME/Types.pm \ -M PerlIO::encoding ... (use forward slashes even under Windows) Cheers, Roderich

Sat Dec 21 06:52:55 2013 RSCHUPP [...] cpan.org - Status changed from 'open' to 'resolved'

Mon Dec 23 09:48:56 2013 md5159 [...] mclink.it - Correspondence added

Subject:	Re: [rt.cpan.org #91530] Missing MIME\types.db from pp-generated code
Date:	Mon, 23 Dec 2013 15:48:49 +0100
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	ric <md5159 [...] mclink.it>

Thanks Roderich, for the detailed explanation and the workaround. Unfortunately, the workaround does not work as expected: despite your guidelines, there's still no trace of the missing types.db, neither within the compiled .exe, nor in the cache. Is there any other chance to force it where it should be? TIA ric On 12/21/2013 12:52 PM, Roderich Schupp via RT wrote: Show quoted text

> So the whole workaround is pp -a > "/path/to/your/MIME/types.db;lib/MIME/types.db" -F > PatchContent=MIME/Types.pm -M PerlIO::encoding ... > Cheers, Roderich

Wed Dec 25 15:04:31 2013 RSCHUPP [...] cpan.org - Correspondence added

Am 2013-12-23 09:48:56, md5159@mclink.it schrieb: Show quoted text

> Unfortunately, the workaround does not work as expected: despite your > guidelines, there's still no trace of the missing types.db, neither > within the compiled .exe, nor in the cache.

Check your -a option for pp. For my Strawberry Perl (installed in C:\strawberry) the following works: pp -a "C:\strawberry\perl\vendor\lib\MIMEtypes.db;lib/MIME/types.db" ... Cheers, Roderich