Bug #127744 for Test-Smoke: smokecurrent.skiptests causes failures in t/porting/manifest.t

> Entering a test file in smokecurrent.skiptests appears to cause failures > in t/porting/manifest.t and perhaps other files. > > Today I smoke-tested OpenBSD-6.4. My instance of that OS is a stock > virtualbox VM provided at vagrantup.com. Like many such boxes, when I > run the Perl 5 test suite or smoke test, I get a higher than normal > incidence of test failures attributable to resource constraints. So > today I tried something new: I made an entry to skip one test file in > smokecurrent.skiptests (and a corresponding entry in smokecurrent.usernote): > > ##### > [p5smoke] 45 $ cat smokecurrent.skiptests > > # One test name on a line > cpan/Sys-Syslog/t/syslog.t > > [p5smoke] 46 $ cat smokecurrent.usernote > cpan/Sys-Syslog/t/syslog.t has too many intermittent > false negatives. Skipping on this platform. See > https://rt.cpan.org/Ticket/Display.html?id=127454. > ##### > > The smoke test report generated was; > > ##### > http://perl5.test-smoke.org/report/74401 > ##### > > cpan/Sys-Syslog/t/syslog.t was apparently skipped -- but > t/porting/manifest.t FAILed in every one of the 8 cells of the smoke test. > > ##### > v5.29.4-93-g8f6aacbbe9 > ---------------------- ---------------------------------------------------- > F F F F (*1) > F F F F -Duseithreads (*1) > | | | | > | | | +------------ perlio DEBUGGING > | | +--------------- stdio DEBUGGING > | +------------------ perlio > +--------------------- stdio > > Applied patches: > SMOKE8f6aacbbe92285ffc82603b952f356914d751df0 > > Skipped tests: > # One test name on a line > cpan/Sys-Syslog/t/syslog.t > > Test failures: > ~~ ../t/porting/manifest.t ..................................... FAILED > 10364 > [stdio/perlio] > [stdio/perlio] -Duseithreads DEBUGGING > [stdio/perlio] DEBUGGING > [stdio/perlio] -Duseithreads > ~~ ../cpan/ExtUtils-Install/t/Installed.t ...................... FAILED > Non-zero exit status: 2 Bad plan. You planned 73 tests but ran 24. > [stdio] -Duseithreads DEBUGGING > > ... > > cpan/Sys-Syslog/t/syslog.t has too many intermittent > false negatives. Skipping on this platform. See > https://rt.cpan.org/Ticket/Display.html?id=127454. > ##### > > Note also that cpan/ExtUtils-Install/t/Installed.t terminated > prematurely as well. > > I went to the perl-current/ directory and manually ran the two test > files through the harness several times. They consistently PASSed. > > I then examined the corresponding log file; see excerpt attached. The > failures in t/porting/manifest.t are clearly associated with the > presence of an entry in smokecurrent.skiptests: > > ##### > # Failed test 10364 - git ls-files gives the same number of files as > MANIFEST lists at porting/manifest.t line 95 > # got "6117" > # expected "6116" > # Failed test 10366 - Nothing added to the repo that isn't in MANIFEST > at porting/manifest.t line 107 > # got "1" > # expected "0" > # Failed test 10367 - Nothing added to the repo that isn't in MANIFEST > at porting/manifest.t line 108 > # got "not in MANIFEST: cpan/Sys-Syslog/t/syslog.t" > # expected "not in MANIFEST: " > # parser guessed wrong encoding expected 'CP1252' got 'UTF-8' > # Will use Digest::MD5 > *** Error 3 in /home/jkeenan/p5smoke/perl-current (makefile:829 > 'test_harness') > > error while running harness target 'test_harness': 1 at > /home/jkeenan/p5smoke/lib/Test/Smoke/App/RunSmoke.pm line 118. > [2018-11-19 16:47:37-0500] [known failed test] > ../t/porting/manifest.t..................................... > [2018-11-19 16:47:38-0500] > [2018-11-19 16:47:37-0500] [known failed test] > ../t/porting/manifest.t..................................... > [2018-11-19 16:47:38-0500] > [2018-11-19 16:47:38-0500] > ../t/porting/manifest.t.....................................FAILED > [2018-11-19 16:47:38-0500] 10364, 10366-10367 > ##### > > Note that while the first failing test number listed is 10364, in a > normal run of this file that would be 10365. > > ##### > ok 10363 - Line 6117 doesn't start with a blank > ok 10364 - File zaphod32_hash.h exists > ok 10365 - MANIFEST sorted properly > ok 10366 - git ls-files gives the same number of files as MANIFEST lists > ok 10367 - no duplicate files in MANIFEST > ok 10368 - Nothing added to the repo that isn't in MANIFEST > ok 10369 - Nothing added to the repo that isn't in MANIFEST > ok 10370 - Nothing in the MANIFEST that isn't tracked by git > ok 10371 - Nothing in the MANIFEST that isn't tracked by git > ##### > > I then went to the Test-Smoke website matrix for failing tests and > examined > http://perl5.test-smoke.org/submatrix?test=../t/porting/manifest.t. I > spot-checked the reports for t/porting/manifest.t. Every one that I > looked at had an entry in smokecurrent.skiptests. See, e.g., > http://perl5.test-smoke.org/report/70865 (from tester abeltje!): > > ##### > Skipped tests: > t/porting/pending-author.t > > Test failures: > ~~ ../t/porting/test_bootstrap.t ............................... FAILED > Non-zero exit status: 2 > [stdio/perlio/locale:nl_NL.utf8] -Dcc='ccache gcc' > [stdio/perlio/locale:nl_NL.utf8] -Dcc='ccache gcc' -Duseithreads > DEBUGGING > [stdio/perlio/locale:nl_NL.utf8] -Dcc='ccache gcc' DEBUGGING > [stdio/perlio/locale:nl_NL.utf8] -Dcc='ccache gcc' -Duseithreads > ~~ ../t/porting/manifest.t ..................................... FAILED 9824 > [stdio/perlio/locale:nl_NL.utf8] -Dcc='ccache gcc' > [stdio/perlio/locale:nl_NL.utf8] -Dcc='ccache gcc' -Duseithreads > DEBUGGING > [stdio/perlio/locale:nl_NL.utf8] -Dcc='ccache gcc' DEBUGGING > [stdio/perlio/locale:nl_NL.utf8] -Dcc='ccache gcc' -Duseithreads > ##### > > (IMO, there is no need to skip t/porting/pending-author.t. In a normal > run of 'make test_harness', it is skipped anyway with message, > "porting/pending-author.t .. skipped: No pending changes".) > > So as it currently stands, using 'smokecurrent.skiptests', rather than > avoiding false negatives, creates new ones. > > Can you investigate? >