Bug #80344 for DBD-SQLite: DBD::SQLite Makefile.PL suggestion

Mon Oct 22 18:58:44 2012 SBarnsle [...] dmc.org - Ticket created

Subject:	DBD::SQLite Makefile.PL suggestion
Date:	Mon, 22 Oct 2012 18:58:30 -0400
To:	"bug-DBD-SQLite [...] rt.cpan.org" <bug-DBD-SQLite [...] rt.cpan.org>
From:	"Barnsley, Steve" <SBarnsle [...] dmc.org>

While porting several perl scripts that do some fairly intense SQLite processing from a Solaris platform to a brand new AIX7 platform I kept running into 'DBD::SQLite::db do failed: disk I/O error at ....' problems. These errors occurred often where $sqlite->do('analyse') occur, but I suspect in other places also. This didn't seem to be a repeatable error, it happened often but not every time for each script. I took a look at the sqlite3.c file in the DBD-SQLite-1.37 distribution and found the following: /* Also fsync the directory containing the file if the DIRSYNC flag ** is set. This is a one-time occurrance. Many systems (examples: AIX) ** are unable to fsync a directory, so ignore errors on the fsync. */ I didn't have time to follow the logic to make sure that the code always ignores the errors for every scenario so I added: if ($^O eq 'aix') { push @CC_DEFINE, '-DSQLITE_DISABLE_DIRSYNC'; } to the Makefile.PL and re-installed DBD::SQLite. Disabling the fsync for directories solved the problem for me so I figured that you might want to consider adding the above line to the Makefile.PL for the next release. Cheers, Steve Barnsley. System details: AIX lawqa-app 1 7 00F785A24C00 powerpc AIX Summary of my perl5 (revision 5 version 16 subversion 0) configuration: Platform: osname=aix, osvers=7.1.0.0, archname=aix-thread-multi-64all uname='aix lawdev-app 1 7 00f785a24c00 powerpc aix ' config_args='-d -Dcc=cc_r -Duseshrplib -Dusethreads -Duse64bitall -Dprefix=/lawson/local/perl' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc_r -q64', ccflags ='-D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=-1 -qnoansialias -DUSE_NATIVE_DLOPEN -DNEED_PTHREAD_INIT -q64 -DUSE_64_BIT_ALL -q64', optimize='-O', cppflags='-D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX_SOURCE -DUSE_NATIVE_DLOPEN -DNEED_PTHREAD_INIT' ccversion='11.1.0.10', gccversion='', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=87654321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='ld', ldflags ='-brtl -bdynamic -b64' libpth=/lib /usr/lib /usr/ccs/lib /usr/lib64 libs=-lbind -lnsl -ldbm -ldl -lld -lm -lcrypt -lpthreads -lc perllibs=-lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads -lc libc=/lib/libc.a, so=a, useshrplib=true, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -bE:/lawson/local/perl/lib/5.16.0/aix-thread-multi-64all/CORE/perl.exp' cccdlflags=' ', lddlflags='-b64 -bhalt:4 -G -bI:$(PERL_INC)/perl.exp -bE:$(BASEEXT).exp -bnoentry -lpthreads -lc -lm' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API Built under aix Compiled at May 24 2012 18:22:47 %ENV: PERL_HOME="/lawson/local/perl" @INC: /lawson/local/perl/lib/site_perl/5.16.0/aix-thread-multi-64all /lawson/local/perl/lib/site_perl/5.16.0 /lawson/local/perl/lib/5.16.0/aix-thread-multi-64all /lawson/local/perl/lib/5.16.0 .

Tue Oct 23 22:34:31 2012 ishigaki [...] cpan.org - Correspondence added

Thanks. Applied to the trunk. On Tue Oct 23 07:58:44 2012, SBarnsle@dmc.org wrote: Show quoted text

> > While porting several perl scripts that do some fairly intense > SQLite processing from a Solaris platform to a brand new AIX7 > platform I kept running into 'DBD::SQLite::db do failed: disk I/O > error at ....' problems. These errors occurred often where > $sqlite->do('analyse') occur, but I suspect in other places also. > This didn't seem to be a repeatable error, it happened often but > not every time for each script. > > I took a look at the > sqlite3.c file in the DBD-SQLite-1.37 distribution and found the > following: > > /* Also fsync the directory containing the file if > the DIRSYNC flag > ** is set. This is a one-time occurrance. > Many systems (examples: AIX) > ** are unable to fsync a directory, > so ignore errors on the fsync. > */ > > I didn't have time to > follow the logic to make sure that the code always ignores the > errors for every scenario so I added: > > if ($^O eq 'aix') { push > @CC_DEFINE, '-DSQLITE_DISABLE_DIRSYNC'; } > > to the Makefile.PL and > re-installed DBD::SQLite. > > Disabling the fsync for directories > solved the problem for me so I figured that you might want to > consider adding the above line to the Makefile.PL for the next > release. > > Cheers, > Steve Barnsley. > > System details: > > AIX > lawqa-app 1 7 00F785A24C00 powerpc AIX > > Summary of my perl5 > (revision 5 version 16 subversion 0) configuration: > > Platform: > osname=aix, osvers=7.1.0.0, archname=aix-thread-multi-64all > uname='aix lawdev-app 1 7 00f785a24c00 powerpc aix ' > config_args='-d -Dcc=cc_r -Duseshrplib -Dusethreads -Duse64bitall > -Dprefix=/lawson/local/perl' > hint=recommended, useposix=true, > d_sigaction=define > useithreads=define, usemultiplicity=define > useperlio=define, d_sfio=undef, uselargefiles=define, > usesocks=undef > use64bitint=define, use64bitall=define, > uselongdouble=undef > usemymalloc=n, bincompat5005=undef > Compiler: > cc='cc_r -q64', ccflags ='-D_ALL_SOURCE > -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=-1 -qnoansialias > -DUSE_NATIVE_DLOPEN -DNEED_PTHREAD_INIT -q64 -DUSE_64_BIT_ALL > -q64', > optimize='-O', > cppflags='-D_ALL_SOURCE > -D_ANSI_C_SOURCE -D_POSIX_SOURCE -DUSE_NATIVE_DLOPEN > -DNEED_PTHREAD_INIT' > ccversion='11.1.0.10', gccversion='', > gccosandvers='' > intsize=4, longsize=8, ptrsize=8, > doublesize=8, byteorder=87654321 > d_longlong=define, > longlongsize=8, d_longdbl=define, longdblsize=8 > ivtype='long', > ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 > alignbytes=8, prototype=define > Linker and Libraries: > ld='ld', ldflags ='-brtl -bdynamic -b64' > libpth=/lib /usr/lib > /usr/ccs/lib /usr/lib64 > libs=-lbind -lnsl -ldbm -ldl -lld -lm > -lcrypt -lpthreads -lc > perllibs=-lbind -lnsl -ldl -lld -lm > -lcrypt -lpthreads -lc > libc=/lib/libc.a, so=a, > useshrplib=true, libperl=libperl.a > gnulibc_version='' > Dynamic Linking: > dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, > ccdlflags=' > -bE:/lawson/local/perl/lib/5.16.0/aix-thread-multi-64all/CORE/

perl.exp' Show quoted text

> cccdlflags=' ', lddlflags='-b64 -bhalt:4 -G > -bI:$(PERL_INC)/perl.exp -bE:$(BASEEXT).exp -bnoentry -lpthreads > -lc -lm' > > > Characteristics of this binary (from libperl): > Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS > PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT > PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT_ALL > USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES > USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE > USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF > USE_REENTRANT_API > Built under aix > Compiled at May 24 2012 > 18:22:47 > %ENV: > PERL_HOME="/lawson/local/perl" > @INC: > /lawson/local/perl/lib/site_perl/5.16.0/aix-thread-multi-64all > /lawson/local/perl/lib/site_perl/5.16.0 > /lawson/local/perl/lib/5.16.0/aix-thread-multi-64all > /lawson/local/perl/lib/5.16.0 > .

Tue Oct 23 22:34:32 2012 The RT System itself - Status changed from 'new' to 'open'

Tue Oct 23 22:34:32 2012 ishigaki [...] cpan.org - Status changed from 'open' to 'patched'

Wed May 29 11:58:11 2013 ishigaki [...] cpan.org - Correspondence added

Closed as DBD::SQLite 1.38_02 was out. Thanks. On Wed Oct 24 11:34:31 2012, ISHIGAKI wrote: Show quoted text

> Thanks. Applied to the trunk. > > On Tue Oct 23 07:58:44 2012, SBarnsle@dmc.org wrote:

> > > > While porting several perl scripts that do some fairly intense > > SQLite processing from a Solaris platform to a brand new AIX7 > > platform I kept running into 'DBD::SQLite::db do failed: disk I/O > > error at ....' problems. These errors occurred often where > > $sqlite->do('analyse') occur, but I suspect in other places also. > > This didn't seem to be a repeatable error, it happened often but > > not every time for each script. > > > > I took a look at the > > sqlite3.c file in the DBD-SQLite-1.37 distribution and found the > > following: > > > > /* Also fsync the directory containing the file if > > the DIRSYNC flag > > ** is set. This is a one-time occurrance. > > Many systems (examples: AIX) > > ** are unable to fsync a directory, > > so ignore errors on the fsync. > > */ > > > > I didn't have time to > > follow the logic to make sure that the code always ignores the > > errors for every scenario so I added: > > > > if ($^O eq 'aix') { push > > @CC_DEFINE, '-DSQLITE_DISABLE_DIRSYNC'; } > > > > to the Makefile.PL and > > re-installed DBD::SQLite. > > > > Disabling the fsync for directories > > solved the problem for me so I figured that you might want to > > consider adding the above line to the Makefile.PL for the next > > release. > > > > Cheers, > > Steve Barnsley. > > > > System details: > > > > AIX > > lawqa-app 1 7 00F785A24C00 powerpc AIX > > > > Summary of my perl5 > > (revision 5 version 16 subversion 0) configuration: > > > > Platform: > > osname=aix, osvers=7.1.0.0, archname=aix-thread-multi-64all > > uname='aix lawdev-app 1 7 00f785a24c00 powerpc aix ' > > config_args='-d -Dcc=cc_r -Duseshrplib -Dusethreads -Duse64bitall > > -Dprefix=/lawson/local/perl' > > hint=recommended, useposix=true, > > d_sigaction=define > > useithreads=define, usemultiplicity=define > > useperlio=define, d_sfio=undef, uselargefiles=define, > > usesocks=undef > > use64bitint=define, use64bitall=define, > > uselongdouble=undef > > usemymalloc=n, bincompat5005=undef > > Compiler: > > cc='cc_r -q64', ccflags ='-D_ALL_SOURCE > > -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=-1 -qnoansialias > > -DUSE_NATIVE_DLOPEN -DNEED_PTHREAD_INIT -q64 -DUSE_64_BIT_ALL > > -q64', > > optimize='-O', > > cppflags='-D_ALL_SOURCE > > -D_ANSI_C_SOURCE -D_POSIX_SOURCE -DUSE_NATIVE_DLOPEN > > -DNEED_PTHREAD_INIT' > > ccversion='11.1.0.10', gccversion='', > > gccosandvers='' > > intsize=4, longsize=8, ptrsize=8, > > doublesize=8, byteorder=87654321 > > d_longlong=define, > > longlongsize=8, d_longdbl=define, longdblsize=8 > > ivtype='long', > > ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 > > alignbytes=8, prototype=define > > Linker and Libraries: > > ld='ld', ldflags ='-brtl -bdynamic -b64' > > libpth=/lib /usr/lib > > /usr/ccs/lib /usr/lib64 > > libs=-lbind -lnsl -ldbm -ldl -lld -lm > > -lcrypt -lpthreads -lc > > perllibs=-lbind -lnsl -ldl -lld -lm > > -lcrypt -lpthreads -lc > > libc=/lib/libc.a, so=a, > > useshrplib=true, libperl=libperl.a > > gnulibc_version='' > > Dynamic Linking: > > dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, > > ccdlflags=' > > -bE:/lawson/local/perl/lib/5.16.0/aix-thread-multi-64all/CORE/

> perl.exp'

> > cccdlflags=' ', lddlflags='-b64 -bhalt:4 -G > > -bI:$(PERL_INC)/perl.exp -bE:$(BASEEXT).exp -bnoentry -lpthreads > > -lc -lm' > > > > > > Characteristics of this binary (from libperl): > > Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS > > PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT > > PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT_ALL > > USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES > > USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE > > USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF > > USE_REENTRANT_API > > Built under aix > > Compiled at May 24 2012 > > 18:22:47 > > %ENV: > > PERL_HOME="/lawson/local/perl" > > @INC: > > /lawson/local/perl/lib/site_perl/5.16.0/aix-thread-multi-64all > > /lawson/local/perl/lib/site_perl/5.16.0 > > /lawson/local/perl/lib/5.16.0/aix-thread-multi-64all > > /lawson/local/perl/lib/5.16.0 > > .

> >

Wed May 29 11:58:11 2013 ishigaki [...] cpan.org - Status changed from 'patched' to 'resolved'