Bug #125932 for Test-Smoke: logs/smokecurrent/*.jsn file and web page not identifying all failing unit tests

Wed Jul 25 15:48:56 2018 jkeenan [...] cpan.org - Ticket created

Subject:

logs/smokecurrent/*.jsn file and web page not identifying all failing unit tests

Here is another case where there is a significant discrepancy between the data generated locally by a Test::Smoke run and what gets displayed on test-smoke.org. This discrepancy may be due to a discrepancy in the logs/smokecurrent/* files on disk. The web report in question is: http://perl5.test-smoke.org/report/68034 (Note: the branch is correctly identified as "smoke-me/khw-sisyphus".) On that page, the failing tests are identified as: ##### Test failures: ~~ ../lib/locale.t ............................................. FAILED 436-437 ##### So, looking at this you would think that only 2 unit tests in lib/locale.t FAILed. But when I examine the log file I find that in each of the 36 configurations *5* unit tests failed in lib/locale.t. ##### [smokecurrent] $ grep -c -E 'lib/locale.*FAILED' logd3bb03e0842cd3214922ab134a48314ee3fe2077.log 36 [smokecurrent] $ grep -n -A1 -E 'lib/locale.*FAILED' logd3bb03e0842cd3214922ab134a48314ee3fe2077.log |head -n 11 8560:[2018-07-25 01:57:20+0000] ../lib/locale.t.............................................FAILED 8561-[2018-07-25 01:57:20+0000] 436-437, 441, 444, 458 -- 8578:[2018-07-25 02:01:59+0000] ../lib/locale.t.............................................FAILED 8579-[2018-07-25 02:01:59+0000] 436-437, 441, 444, 458 -- 9311:[2018-07-25 02:12:55+0000] ../lib/locale.t.............................................FAILED 9312-[2018-07-25 02:12:55+0000] 436-437, 441, 444, 458 -- 9329:[2018-07-25 02:18:20+0000] ../lib/locale.t.............................................FAILED 9330-[2018-07-25 02:18:20+0000] 436-437, 441, 444, 458 ##### The same results are recorded in the corresponding 'out' file. ##### $ grep -c -E 'lib/locale.*FAILED' outd3bb03e0842cd3214922ab134a48314ee3fe2077.out 36 [smokecurrent] $ grep -n -A1 'lib/locale.*FAILED' outd3bb03e0842cd3214922ab134a48314ee3fe2077.out |head -n 11 19:../lib/locale.t.............................................FAILED 20- 436-437, 441, 444, 458 -- 24:../lib/locale.t.............................................FAILED 25- 436-437, 441, 444, 458 -- 36:../lib/locale.t.............................................FAILED 37- 436-437, 441, 444, 458 -- 41:../lib/locale.t.............................................FAILED 42- 436-437, 441, 444, 458 ##### The same results are recorded in the corresponding 'rpt' file. ##### Tests skipped on user request: # One test name on a line Failures: (common-args) -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing" [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" -Duse64bitint [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused-arguments" -Duse64bitint [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" -Duselongdouble [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused-arguments" -Duselongdouble [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" -Dusemorebits [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused-arguments" -Dusemorebits [stdio/perlio] -Duseithreads -Dcc="g++" -Duse64bitint [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Duse64bitint [stdio/perlio] -Duseithreads -Dcc="g++" -Duselongdouble [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Duselongdouble [stdio/perlio] -Duseithreads -Dcc="g++" -Dusemorebits [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Dusemorebits [stdio/perlio] -Duseithreads -Dcc="gcc" -Duse64bitint [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Duse64bitint [stdio/perlio] -Duseithreads -Dcc="gcc" -Duselongdouble [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Duselongdouble [stdio/perlio] -Duseithreads -Dcc="gcc" -Dusemorebits [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Dusemorebits ../lib/locale.t.............................................FAILED 436-437, 441, 444, 458 ##### However, when I look at the corresponding 'jsn' file, I see that only 2 of the 5 failing unit tests are being identified. The following block appears 36 times in jsnd3bb03e0842cd3214922ab134a48314ee3fe2077.jsn. ##### "failures" : [ { "status" : "FAILED", "extra" : [ "436-437" ], "test" : "../lib/locale.t" } ], ##### The failures in unit tests 441, 444, 458 in lib/locale.t never make it into the .jsn file. Am I correct in thinking that the web page is composed from the .jsn file? If so, then the problem is in the composition of the .jsn file on disk. If not, then test-smoke.org is incorrectly processing whatever file it does get. Do you know why this is occurring? Thank you very much. Jim Keenan

Wed Jul 25 15:54:23 2018 jkeenan [...] cpan.org - Reference to ticket #90612 added

Wed Aug 15 10:28:36 2018 jkeenan [...] cpan.org - Correspondence added

On Wed Jul 25 15:48:56 2018, JKEENAN wrote: Show quoted text

> Here is another case where there is a significant discrepancy between > the data generated locally by a Test::Smoke run and what gets > displayed on test-smoke.org. This discrepancy may be due to a > discrepancy in the logs/smokecurrent/* files on disk. > > The web report in question is: > > http://perl5.test-smoke.org/report/68034 > > (Note: the branch is correctly identified as "smoke-me/khw-sisyphus".) > > On that page, the failing tests are identified as: > > ##### > Test failures: > ~~ ../lib/locale.t ............................................. > FAILED 436-437 > ##### > > So, looking at this you would think that only 2 unit tests in > lib/locale.t FAILed. But when I examine the log file I find that in > each of the 36 configurations *5* unit tests failed in lib/locale.t. > > ##### > [smokecurrent] $ grep -c -E 'lib/locale.*FAILED' > logd3bb03e0842cd3214922ab134a48314ee3fe2077.log > 36 > [smokecurrent] $ grep -n -A1 -E 'lib/locale.*FAILED' > logd3bb03e0842cd3214922ab134a48314ee3fe2077.log |head -n 11 > 8560:[2018-07-25 01:57:20+0000] > ../lib/locale.t.............................................FAILED > 8561-[2018-07-25 01:57:20+0000] 436-437, 441, 444, 458 > -- > 8578:[2018-07-25 02:01:59+0000] > ../lib/locale.t.............................................FAILED > 8579-[2018-07-25 02:01:59+0000] 436-437, 441, 444, 458 > -- > 9311:[2018-07-25 02:12:55+0000] > ../lib/locale.t.............................................FAILED > 9312-[2018-07-25 02:12:55+0000] 436-437, 441, 444, 458 > -- > 9329:[2018-07-25 02:18:20+0000] > ../lib/locale.t.............................................FAILED > 9330-[2018-07-25 02:18:20+0000] 436-437, 441, 444, 458 > ##### > > The same results are recorded in the corresponding 'out' file. > > ##### > $ grep -c -E 'lib/locale.*FAILED' > outd3bb03e0842cd3214922ab134a48314ee3fe2077.out > 36 > [smokecurrent] $ grep -n -A1 'lib/locale.*FAILED' > outd3bb03e0842cd3214922ab134a48314ee3fe2077.out |head -n 11 > 19:../lib/locale.t.............................................FAILED > 20- 436-437, 441, 444, 458 > -- > 24:../lib/locale.t.............................................FAILED > 25- 436-437, 441, 444, 458 > -- > 36:../lib/locale.t.............................................FAILED > 37- 436-437, 441, 444, 458 > -- > 41:../lib/locale.t.............................................FAILED > 42- 436-437, 441, 444, 458 > ##### > > The same results are recorded in the corresponding 'rpt' file. > > ##### > Tests skipped on user request: > # One test name on a line > Failures: (common-args) -Doptimize="-O2 -pipe -fstack-protector -fno- > strict-aliasing" > [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" > -Duse64bitint > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused- > arguments" -Duse64bitint > [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" > -Duselongdouble > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused- > arguments" -Duselongdouble > [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" > -Dusemorebits > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused- > arguments" -Dusemorebits > [stdio/perlio] -Duseithreads -Dcc="g++" -Duse64bitint > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Duse64bitint > [stdio/perlio] -Duseithreads -Dcc="g++" -Duselongdouble > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Duselongdouble > [stdio/perlio] -Duseithreads -Dcc="g++" -Dusemorebits > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Dusemorebits > [stdio/perlio] -Duseithreads -Dcc="gcc" -Duse64bitint > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Duse64bitint > [stdio/perlio] -Duseithreads -Dcc="gcc" -Duselongdouble > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Duselongdouble > [stdio/perlio] -Duseithreads -Dcc="gcc" -Dusemorebits > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Dusemorebits > ../lib/locale.t.............................................FAILED > 436-437, 441, 444, 458 > ##### > > However, when I look at the corresponding 'jsn' file, I see that only > 2 of the 5 failing unit tests are being identified. The following > block appears 36 times in > jsnd3bb03e0842cd3214922ab134a48314ee3fe2077.jsn. > > ##### > "failures" : [ > { > "status" : "FAILED", > "extra" : [ > "436-437" > ], > "test" : "../lib/locale.t" > } > ], > ##### > > The failures in unit tests 441, 444, 458 in lib/locale.t never make it > into the .jsn file. Am I correct in thinking that the web page is > composed from the .jsn file? If so, then the problem is in the > composition of the .jsn file on disk. If not, then test-smoke.org is > incorrectly processing whatever file it does get. > > Do you know why this is occurring? > > Thank you very much. > Jim Keenan

I have filed a pull request correcting this problem at: https://github.com/abeltje/Test-Smoke/pull/41 Please review and, if satisfactory, apply to master and issue a new CPAN release at your earliest opportunity. Thank you very much. Jim Keenan

Wed Nov 14 12:30:56 2018 jkeenan [...] cpan.org - Correspondence added

On Wed Aug 15 10:28:36 2018, JKEENAN wrote: Show quoted text

> On Wed Jul 25 15:48:56 2018, JKEENAN wrote:

> > Here is another case where there is a significant discrepancy between > > the data generated locally by a Test::Smoke run and what gets > > displayed on test-smoke.org. This discrepancy may be due to a > > discrepancy in the logs/smokecurrent/* files on disk. > > > > The web report in question is: > > > > http://perl5.test-smoke.org/report/68034 > > > > (Note: the branch is correctly identified as "smoke-me/khw- > > sisyphus".) > > > > On that page, the failing tests are identified as: > > > > ##### > > Test failures: > > ~~ ../lib/locale.t ............................................. > > FAILED 436-437 > > ##### > > > > So, looking at this you would think that only 2 unit tests in > > lib/locale.t FAILed. But when I examine the log file I find that in > > each of the 36 configurations *5* unit tests failed in lib/locale.t. > > > > ##### > > [smokecurrent] $ grep -c -E 'lib/locale.*FAILED' > > logd3bb03e0842cd3214922ab134a48314ee3fe2077.log > > 36 > > [smokecurrent] $ grep -n -A1 -E 'lib/locale.*FAILED' > > logd3bb03e0842cd3214922ab134a48314ee3fe2077.log |head -n 11 > > 8560:[2018-07-25 01:57:20+0000] > > ../lib/locale.t.............................................FAILED > > 8561-[2018-07-25 01:57:20+0000] 436-437, 441, 444, 458 > > -- > > 8578:[2018-07-25 02:01:59+0000] > > ../lib/locale.t.............................................FAILED > > 8579-[2018-07-25 02:01:59+0000] 436-437, 441, 444, 458 > > -- > > 9311:[2018-07-25 02:12:55+0000] > > ../lib/locale.t.............................................FAILED > > 9312-[2018-07-25 02:12:55+0000] 436-437, 441, 444, 458 > > -- > > 9329:[2018-07-25 02:18:20+0000] > > ../lib/locale.t.............................................FAILED > > 9330-[2018-07-25 02:18:20+0000] 436-437, 441, 444, 458 > > ##### > > > > The same results are recorded in the corresponding 'out' file. > > > > ##### > > $ grep -c -E 'lib/locale.*FAILED' > > outd3bb03e0842cd3214922ab134a48314ee3fe2077.out > > 36 > > [smokecurrent] $ grep -n -A1 'lib/locale.*FAILED' > > outd3bb03e0842cd3214922ab134a48314ee3fe2077.out |head -n 11 > > 19:../lib/locale.t.............................................FAILED > > 20- 436-437, 441, 444, 458 > > -- > > 24:../lib/locale.t.............................................FAILED > > 25- 436-437, 441, 444, 458 > > -- > > 36:../lib/locale.t.............................................FAILED > > 37- 436-437, 441, 444, 458 > > -- > > 41:../lib/locale.t.............................................FAILED > > 42- 436-437, 441, 444, 458 > > ##### > > > > The same results are recorded in the corresponding 'rpt' file. > > > > ##### > > Tests skipped on user request: > > # One test name on a line > > Failures: (common-args) -Doptimize="-O2 -pipe -fstack-protector -fno- > > strict-aliasing" > > [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" > > -Duse64bitint > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused- > > arguments" -Duse64bitint > > [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" > > -Duselongdouble > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused- > > arguments" -Duselongdouble > > [stdio/perlio] -Duseithreads -Dcc="clang -Qunused-arguments" > > -Dusemorebits > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="clang -Qunused- > > arguments" -Dusemorebits > > [stdio/perlio] -Duseithreads -Dcc="g++" -Duse64bitint > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Duse64bitint > > [stdio/perlio] -Duseithreads -Dcc="g++" -Duselongdouble > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Duselongdouble > > [stdio/perlio] -Duseithreads -Dcc="g++" -Dusemorebits > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="g++" -Dusemorebits > > [stdio/perlio] -Duseithreads -Dcc="gcc" -Duse64bitint > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Duse64bitint > > [stdio/perlio] -Duseithreads -Dcc="gcc" -Duselongdouble > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Duselongdouble > > [stdio/perlio] -Duseithreads -Dcc="gcc" -Dusemorebits > > [stdio/perlio] -DDEBUGGING -Duseithreads -Dcc="gcc" -Dusemorebits > > ../lib/locale.t.............................................FAILED > > 436-437, 441, 444, 458 > > ##### > > > > However, when I look at the corresponding 'jsn' file, I see that only > > 2 of the 5 failing unit tests are being identified. The following > > block appears 36 times in > > jsnd3bb03e0842cd3214922ab134a48314ee3fe2077.jsn. > > > > ##### > > "failures" : [ > > { > > "status" : "FAILED", > > "extra" : [ > > "436-437" > > ], > > "test" : "../lib/locale.t" > > } > > ], > > ##### > > > > The failures in unit tests 441, 444, 458 in lib/locale.t never make > > it > > into the .jsn file. Am I correct in thinking that the web page is > > composed from the .jsn file? If so, then the problem is in the > > composition of the .jsn file on disk. If not, then test-smoke.org is > > incorrectly processing whatever file it does get. > > > > Do you know why this is occurring? > > > > Thank you very much. > > Jim Keenan

> > I have filed a pull request correcting this problem at: > > https://github.com/abeltje/Test-Smoke/pull/41 > > Please review and, if satisfactory, apply to master and issue a new > CPAN release at your earliest opportunity. >

Today I am evaluating Test-Smoke-1.72_06. On the basis of a comparison between this smoke test report: http://perl5.test-smoke.org/report/68034 and this smoke-test report: http://perl5.test-smoke.org/report/74020 ... I infer that 1.72_06 addresses the problem. This ticket will be closable once a new production release of Test-Smoke that includes this correction is made. Thank you very much. Jim Keenan