Bug #101883 for BerkeleyDB: Error opening ErrFile with PerlIO

Mon Feb 02 11:02:24 2015 david [...] cobite.com - Ticket created

Subject:	Error opening ErrFile with PerlIO_findFILE
Date:	Mon, 02 Feb 2015 11:02:01 -0500
To:	bug-BerkeleyDB [...] rt.cpan.org
From:	David Mansfield <david [...] cobite.com>

We have stumbled across an issue (not sure whether it is actually a bug or not!) pertaining to the opening of the ErrFile in the XS code. In our case, we have duped a pipe's write filehandle to replace STDOUT and STDERR, and we're passing: -ErrFile => *STDERR It reduces to the fact that PerlIO_findFILE is returning NULL in the GetFILEptr macro (and USE_PERLIO is defined). Breaking it down during debugging, the pointer returned by the inner functions, IoIFP(sv_2io(sv)), is valid, but the above method returns NULL. The "documentation" of the method PerlIO_findFILE says: "Returns previously 'exported' FILE * (if any)." There is another method on the man page, PerlIO_exportFILE, which is defined as: "Given an PerlIO * return a 'native' FILE * suitable for passing to code expecting to be compiled and linked with ANSI C stdio.h" I tried substituting exportFILE and it is now working - but I'm not sure of the impact. Maybe if findFILE returns NULL, then exportFILE should be tried ? I don't know the semantics for this. As to why this is happening now, I cannot explain it. We have half a dozen systems across different Centos releases running with the exact same code and on a new system, running the exact same versions of everything, this is happening. The only thing about the new system is that it has a lot of CPU cores - so a race condition seems the most likely culprit. I can only assume something is changed about the state of the perlio layers that causes this behavior change, but I cannot figure out what. -- Thanks, David Mansfield Cobite, INC.

Wed Feb 04 10:58:34 2015 david [...] cobite.com - Correspondence added

Subject:	Re: [rt.cpan.org #101883] AutoReply: Error opening ErrFile with PerlIO_findFILE
Date:	Wed, 04 Feb 2015 10:58:20 -0500
To:	bug-BerkeleyDB [...] rt.cpan.org
From:	David Mansfield <david [...] cobite.com>

I have a small script that reproduces this. It seems to be related to Proc::Daemon which closes all filehandles. Basically it comes down to this: If I re-open STDERR as a pipe filehandle, and pass as ErrFile it fails, but if STDERR is closed first, it works. According to perl close documentation, closing first is not required. Script is attached (I'll check that RT processes attachments and put inline if not).

Message body is not shown because sender requested not to inline it.

Sun Feb 15 05:23:06 2015 pmqs [...] cpan.org - Correspondence added

Hey David, what is different on the new system where it is failing? Different version of Perl? sorry for not replying sooner. Paul

Sun Feb 15 05:23:07 2015 The RT System itself - Status changed from 'new' to 'open'

Wed Feb 18 09:26:38 2015 david [...] cobite.com - Correspondence added

Subject:	Re: [rt.cpan.org #101883] Error opening ErrFile with PerlIO_findFILE
Date:	Wed, 18 Feb 2015 09:26:25 -0500
To:	bug-BerkeleyDB [...] rt.cpan.org
From:	David Mansfield <david [...] cobite.com>

On 02/15/2015 05:23 AM, Paul Marquess via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=101883 > > > Hey David, > > what is different on the new system where it is failing? Different version of Perl? > > sorry for not replying sooner. > Paul

I believe it is the newer version of Proc::Daemon that is significant. All other things remained basically constant (or were made consistent during debugging). The working one has perl-Proc-Daemon-0.06-1.el6.noarch and the broken one is perl-Proc-Daemon-0.14-9.el6.noarch. There are significant differences in the way those versions handle existing open file handles while daemonizing. I can't guarantee there's not a bug there, but I don't know the semantics of the perlio level in native code - and I can't rule out a perl bug. After all the difference in the test script between working and not working is a "close STDERR" right before re-opening, but the perldoc for open clearly states it should not be necessary when re-opening. So hopefully you know more about this than me. Thanks, David

Wed Feb 18 15:48:50 2015 pmqs [...] cpan.org - Correspondence added

On Wed Feb 18 09:26:38 2015, david@cobite.com wrote: Show quoted text

> > On 02/15/2015 05:23 AM, Paul Marquess via RT wrote:

> > <URL: https://rt.cpan.org/Ticket/Display.html?id=101883 > > > > > Hey David, > > > > what is different on the new system where it is failing? Different > > version of Perl? > > > > sorry for not replying sooner. > > Paul

> > I believe it is the newer version of Proc::Daemon that is significant.

Version of Perl the same? Show quoted text

> All other things remained basically constant (or were made consistent > during debugging). The working one has > perl-Proc-Daemon-0.06-1.el6.noarch and the broken one is > perl-Proc-Daemon-0.14-9.el6.noarch.

Could you try revering back to the previous version? That would confirm if Proc::Daemon is the problem. Show quoted text

> There are significant differences in the way those versions handle > existing open file handles while daemonizing. I can't guarantee > there's > not a bug there, but I don't know the semantics of the perlio level in > native code - and I can't rule out a perl bug. After all the > difference > in the test script between working and not working is a "close STDERR" > right before re-opening, but the perldoc for open clearly states it > should not be necessary when re-opening. So hopefully you know more > about this than me.

Messing with filehandles is a real pain especially in XS code. Can't say I'm any expert. Anyway, I've run a quick regression test and changing to PerlIO_exportFILE didn't break on any perl + BerkeleyDB all the way back to 5.6.0. I'll run the full test overnight and see how that goes.

Wed Feb 18 16:11:30 2015 david [...] cobite.com - Correspondence added

Subject:	Re: [rt.cpan.org #101883] Error opening ErrFile with PerlIO_findFILE
Date:	Wed, 18 Feb 2015 16:11:15 -0500
To:	bug-BerkeleyDB [...] rt.cpan.org
From:	David Mansfield <david [...] cobite.com>

On 02/18/2015 03:48 PM, Paul Marquess via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=101883 > > > On Wed Feb 18 09:26:38 2015, david@cobite.com wrote:

>> On 02/15/2015 05:23 AM, Paul Marquess via RT wrote:

>>> <URL: https://rt.cpan.org/Ticket/Display.html?id=101883 > >>> >>> Hey David, >>> >>> what is different on the new system where it is failing? Different >>> version of Perl? >>> >>> sorry for not replying sooner. >>> Paul

>> I believe it is the newer version of Proc::Daemon that is significant.

> Version of Perl the same? >

>> All other things remained basically constant (or were made consistent >> during debugging). The working one has >> perl-Proc-Daemon-0.06-1.el6.noarch and the broken one is >> perl-Proc-Daemon-0.14-9.el6.noarch.

> Could you try revering back to the previous version? That would confirm if Proc::Daemon is the problem. >

>> There are significant differences in the way those versions handle >> existing open file handles while daemonizing. I can't guarantee >> there's >> not a bug there, but I don't know the semantics of the perlio level in >> native code - and I can't rule out a perl bug. After all the >> difference >> in the test script between working and not working is a "close STDERR" >> right before re-opening, but the perldoc for open clearly states it >> should not be necessary when re-opening. So hopefully you know more >> about this than me.

> Messing with filehandles is a real pain especially in XS code. Can't say I'm any expert. > > Anyway, I've run a quick regression test and changing to PerlIO_exportFILE didn't break on any perl + BerkeleyDB all the way back to 5.6.0. > > I'll run the full test overnight and see how that goes. >

In the complete knowledge void I live in regarding this issue, I would do if PerlIO_findFILE returns NULL, try PerlIO_exportFILE. Seems most defensive and I would worry about possibly leaking if the allocation doesn't need to be done, or aliasing an existing handle and having buffering issues. Again, the sum total of what I know is what you see above :-( Thanks, David

Sun Feb 22 05:58:35 2015 pmqs [...] cpan.org - Correspondence added

Hey David, BerkeleyDB-0.55 just uploaed to CPAN. Ping me if you have any problems. cheers Paul

Bug #101883 for BerkeleyDB: Error opening ErrFile with PerlIO_findFILE