Bug #99880 for Crypt-Random: Dubious test in t/generator.t

Wed Oct 29 07:20:18 2014 paul [...] city-fan.org - Ticket created

Subject:

Dubious test in t/generator.t

One of the tests in t/generator.t involves generating two random 10-bit integers and checking that they are not the same. The test fails if they are the same, which would normally happen about 0.1% of the time. This is probably not a big problem but for downstream distributions that do frequent rebuilds, this can trigger a "fails to build from source" bug that then needs to be investigated.

Fri Sep 25 15:22:51 2015 SREZIC [...] cpan.org - Correspondence added

On 2014-10-29 07:20:18, paul@city-fan.org wrote: Show quoted text

> One of the tests in t/generator.t involves generating two random 10- > bit integers and checking that they are not the same. The test fails > if they are the same, which would normally happen about 0.1% of the > time. This is probably not a big problem but for downstream > distributions that do frequent rebuilds, this can trigger a "fails to > build from source" bug that then needs to be investigated.

This probably explains the fails visible at CPAN Testers: http://matrix.cpantesters.org/?dist=Crypt-Random%201.25 Currently it's: * PASS: 6801 * FAIL: 48

Fri Sep 25 15:22:51 2015 The RT System itself - Status changed from 'new' to 'open'

Sun Oct 01 00:10:48 2017 KENTNL [...] cpan.org - Correspondence added

replaced the == test with

warn("$x == $y Strength=$$gen{Strength} Provider=$$gen{Provider}") if $x == $y;

And ran 10,000 times. ( == 30,000 invocations of test() )

67/30000 = 0.22%

19 were from devurandom ( 0.19% )
24 were from devrandom ( 0.24% )
24 were from rand ( 0.24% )

However, the *distribution* of where these pairs occured were a bit disturbing: 100% of them occurred above 512

Which suggests there's a whole bit of data lost somewhere, or some kind of leak in the random.

Image of distribution, and the data of sequential collisions attached.

This seems potentially far more serious than a bad test.

--
- CPAN kentnl@cpan.org
- Gentoo Perl Maintainer kentnl@gentoo.org ( perl@gentoo.org )

Subject:

samples.tsv

rand devrandom devurandom 584 528 527 587 529 544 607 559 581 615 597 593 617 599 599 620 600 621 637 601 666 670 603 677 714 610 701 727 679 726 732 737 835 766 761 858 772 808 883 808 821 884 815 829 911 816 855 915 817 911 964 818 913 987 822 933 1011 831 960 850 992 867 994 890 1009 937 1011

Subject:

snapshot_20171001_170650.png

Download snapshot_20171001_170650.png
image/png 19.6k

Sun Oct 01 04:18:48 2017 KENTNL [...] cpan.org - Correspondence added

Carefully reading the documentation says that this sort of distribution is intended *design*. Just from a consumer perspective is incredibly unexpected, and very poorly documented.

Even conceptually, I find this design questionable, but I'll have to follow up with some people to see how reasonable it is.

--
- CPAN kentnl@cpan.org
- Gentoo Perl Maintainer kentnl@gentoo.org ( perl@gentoo.org )

Sat Dec 22 11:12:33 2018 VIPUL [...] cpan.org - Correspondence added

I am fixing this in upcoming release. I don't think we can keep this particular test for the reason you describe. Even at high bitsizes, it is not guaranteed to pass. On Wed Oct 29 07:20:18 2014, paul@city-fan.org wrote: Show quoted text

> One of the tests in t/generator.t involves generating two random 10- > bit integers and checking that they are not the same. The test fails > if they are the same, which would normally happen about 0.1% of the > time. This is probably not a big problem but for downstream > distributions that do frequent rebuilds, this can trigger a "fails to > build from source" bug that then needs to be investigated.

Sat Dec 22 11:28:11 2018 VIPUL [...] cpan.org - Correspondence added

Looks worrisome. I'll dig into it. On Sun Oct 01 00:10:48 2017, KENTNL wrote: Show quoted text

> replaced the == test with > > warn("$x == $y Strength=$$gen{Strength} Provider=$$gen{Provider}") if > $x == $y; > > And ran 10,000 times. ( == 30,000 invocations of test() ) > > 67/30000 = 0.22% > > 19 were from devurandom ( 0.19% ) > 24 were from devrandom ( 0.24% ) > 24 were from rand ( 0.24% ) > > However, the *distribution* of where these pairs occured were a bit > disturbing: > 100% of them occurred above 512 > > Which suggests there's a whole bit of data lost somewhere, or some > kind of leak > in the random. > > Image of distribution, and the data of sequential collisions attached. > > This seems potentially far more serious than a bad test. > > -- > - CPAN kentnl@cpan.org > - Gentoo Perl Maintainer kentnl@gentoo.org ( perl@gentoo.org )

Sat Dec 22 13:59:00 2018 VIPUL [...] cpan.org - Correspondence added

Probably not a great decision at the time, but uniform distribution over the interval requires passing Uniform => 1 to integer(), makerandom() etc. In 1.52, I allow Uniform => 1 to be passed to the constructor and also introduced a chi-square test to test a generated distribution. I am going to further test dependent modules and see if I can safely make Uniform => 1 the default. On Sat Dec 22 11:28:11 2018, VIPUL wrote: Show quoted text

> Looks worrisome. I'll dig into it. > > On Sun Oct 01 00:10:48 2017, KENTNL wrote:

> > replaced the == test with > > > > warn("$x == $y Strength=$$gen{Strength} Provider=$$gen{Provider}") if > > $x == $y; > > > > And ran 10,000 times. ( == 30,000 invocations of test() ) > > > > 67/30000 = 0.22% > > > > 19 were from devurandom ( 0.19% ) > > 24 were from devrandom ( 0.24% ) > > 24 were from rand ( 0.24% ) > > > > However, the *distribution* of where these pairs occured were a bit > > disturbing: > > 100% of them occurred above 512 > > > > Which suggests there's a whole bit of data lost somewhere, or some > > kind of leak > > in the random. > > > > Image of distribution, and the data of sequential collisions attached. > > > > This seems potentially far more serious than a bad test. > > > > -- > > - CPAN kentnl@cpan.org > > - Gentoo Perl Maintainer kentnl@gentoo.org ( perl@gentoo.org )

>

Sat Dec 22 13:59:00 2018 VIPUL [...] cpan.org - Status changed from 'open' to 'resolved'