Skip Menu |

This queue is for tickets about the Config-General CPAN distribution.

Report information
The Basics
Id: 118383
Status: open
Priority: 0/
Queue: Config-General

People
Owner: Nobody in particular
Requestors: justice__8 [...] hotmail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



CC: "tlinden [...] cpan.org" <tlinden [...] cpan.org>, "valid-until-21-Oct-2016 [...] daemon.de" <valid-until-21-Oct-2016 [...] daemon.de>
Subject: Variables interpolation sometimes doesn't work
Date: Fri, 14 Oct 2016 09:54:51 +0000
To: "bug-Config-General [...] rt.cpan.org" <bug-Config-General [...] rt.cpan.org>
From: JusTiCe 8 <justice__8 [...] hotmail.com>
Hi Thomas, using variables interpolation cause sometimes random failure in the Config::General constructor (probably) caused by Perl hashes randomness. This sample (see at the bottom) run a loop 1000 times and here are some results I got: ok=794 ko=206 ok=673 ko=327 ok=791 ko=209 ok=814 ko=186 ok=999 ko=1 ok=672 ko=328 ok is when vars are correctly interpolated as expected ko is when the constructor raise an exception so you can see here it failed from time to time in a very (almost) unpredictable way. I've done myself a dynamic configuration system and my solution to this issue is to make multiple passes to fill a dictionary made of vars' values, until it no longer contains any vars at all, like this DIC= (after reading config file) var1 = 3 var2 = $var1/$var3 var3 = something on a second pass => var2 = 3/something, dictionary is free of var and we're good to go And so do n passes with a maximum of a given number to avoid infinite loop, raise an error only when there are still unknown variables in config after these n passes. Included is a sample code and a related configuration files. Regards.
Download ConfigGeneralIssue
application/octet-stream 779b

Message body not shown because it is not plain text.

Download .configgen-issue.cfg
application/octet-stream 166b

Message body not shown because it is not plain text.

Hi, I can't reproduce this: % perl ConfigGeneralIssue ok=1000 ko=0 So, can you uncomment this line: #print "CG error: [$@]\n"; and post the output? Also, which version are you using? Add: print "Config::General::Version: $Config::General::VERSION\n"; And which perl version (perl -v)? Thanks, Tom
Subject: Re: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work
Date: Fri, 14 Oct 2016 11:11:26 +0000
To: "bug-Config-General [...] rt.cpan.org" <bug-Config-General [...] rt.cpan.org>
From: JusTiCe 8 <justice__8 [...] hotmail.com>
Oups, my bad, sorry. it's on Debian stretch (current testing): perl 5.24.1 Config::General 2.63 (and for what it worth on a VMware guest) The error msg is: Use of uninitialized variable ($item) while loading config entry: var = $item at ConfigGeneralIssue line 24. (line 24 is the 1st line of the constructor call) How many time did you run it ? I have also got a 1000/0 but it's quite an anomaly on my setup. Show quoted text
________________________________ From: T. Linden via RT <bug-Config-General@rt.cpan.org> Sent: Friday, October 14, 2016 12:21 PM To: justice__8@hotmail.com Subject: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work <URL: https://rt.cpan.org/Ticket/Display.html?id=118383 > Hi, I can't reproduce this: % perl ConfigGeneralIssue ok=1000 ko=0 So, can you uncomment this line: #print "CG error: [$@]\n"; and post the output? Also, which version are you using? Add: print "Config::General::Version: $Config::General::VERSION\n"; And which perl version (perl -v)? Thanks, Tom
Hi, Show quoted text
> How many time did you run it ?
1000 times, no problems. Show quoted text
> (and for what it worth on a VMware guest)
There we have it. Entropy exhausts sometimes very fast on VMs. Retry on metal and you'll see no more problems. You might try to workaround the entropy problem a little, e.g. install egd or stuff like that. Tom
Subject: Re: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work
Date: Fri, 14 Oct 2016 13:36:26 +0000
To: "bug-Config-General [...] rt.cpan.org" <bug-Config-General [...] rt.cpan.org>
From: JusTiCe 8 <justice__8 [...] hotmail.com>
For what it worth, I have installed haveged and, I still got issues with higher available entropy than before. (report of entropy comes from /proc/sys/kernel/random/entropy_avail), cf https://wiki.archlinux.org/index.php/Haveged) ok=681 ko=319 1086 <== this is the entropy ok=1000 ko=0 1108 ok=655 ko=345 1115 ok=808 ko=192 2231 ok=670 ko=330 2236 ok=806 ko=194 2240 ok=801 ko=199 2245 on a Linux physical host (Debian 8.5, kernel 3.16.7-ctk25-2 x64, Perl 5.20.2 and Config::General 2.56), I got similar results: 666 ok=674 ko=326 ok=661 ko=339 ok=791 ko=209 ok=803 ko=197 ok=672 ko=328 669 670 672 ok=661 ko=339 haveged installed => entropy grown to 2463 but ok=664 ko=336 ok=667 ko=333 2308 ok=793 ko=207 2059 ok=657 ko=343 (lonely digits = cat .../entropy_avail) I'm not really convinced. IMHO, as variables can be defined and used mostly anywhere, even using any kind of sort, you can't assume any order between variables declarations and uses, so interpolation cannot be done in a single pass (except when you only allow a single layer of variables, ie bar = $xyz, foo=$bar not allowed).
Show quoted text
> on a Linux physical host (Debian 8.5, kernel 3.16.7-ctk25-2 x64, Perl > 5.20.2 and Config::General 2.56), I got similar results: > > 666 > ok=674 ko=326 > ok=661 ko=339 > ok=791 ko=209 > ok=803 ko=197 > ok=672 ko=328
I still can't reproduce this, not on FreeBSD, not on Arch Linux, not on OpenBSD, not on Cygwin. Show quoted text
> IMHO, as variables can be defined and used mostly anywhere, even using > any kind of sort, you can't assume any order between variables > declarations and uses, so interpolation cannot be done in a single > pass (except when you only allow a single layer of variables, ie bar = > $xyz, foo=$bar not allowed).
I don't understand this sentence. ---- However, I misinterpreted the issue anyway. It doesn't have anything to do with randomness, since Config::General::Interpolate doesn't use random things at all. Sorry for the confusion. So. I've got no idea :(
Subject: Re: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work
Date: Mon, 17 Oct 2016 12:41:44 +0000
To: "bug-Config-General [...] rt.cpan.org" <bug-Config-General [...] rt.cpan.org>
From: JusTiCe 8 <justice__8 [...] hotmail.com>
Show quoted text
> However, I misinterpreted the issue anyway. It doesn't have anything to do with randomness, since Config::General::Interpolate doesn't use random things at all. Sorry for the confusion.
It is, all loop like: for (keys %{$config}) especially in "_var_hash_stacker" and "_clean_stack" subs of Interpolated module get a list of keys in not always the same order each time the test program is run. (quote from keys function doc: "Hash entries are returned in an apparently random order") I have added a call to sort on all loops on keys in Interpolated module, then I got a consistent 500/500 ok/ko result. If I change sorting order to descending (adding "sort { $b cmp $a } " in loops above), I got a consistent 1000/0 result (I'm lucky on this, depending on blocks and variables names, it can also failed all the time). I have also left only one variable and one config item which use it directly: myvar = something /*item = $myvar*/ <blockone> foo = notbar var = $myvar /* $item */ another = novar /* $myvar */ </blockone> it still failed without sorting keys. Attached is the result of keys sorting order with no sorting, sorting descending and sort ascending (default) in the two subs given above. Show quoted text
> I don't understand this sentence.
My bad. I just mean IMHO variable s interpolation in configuration file, using Perl hashes, cannot be done in one single pass while parsing the file, as the usage of a variable can be parsed BEFORE the actual variable declaration, ie: myvar = 1 ... foobar = $myvar can be either parsed in the "assumed" order (myvar before foobar = ok), or the opposite (foobar before myvar = failure) because on how keys and hashes works in Perl. Show quoted text
________________________________ From: T. Linden via RT <bug-Config-General@rt.cpan.org> Sent: Friday, October 14, 2016 9:32 PM To: justice__8@hotmail.com Subject: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work <URL: https://rt.cpan.org/Ticket/Display.html?id=118383 >
> on a Linux physical host (Debian 8.5, kernel 3.16.7-ctk25-2 x64, Perl > 5.20.2 and Config::General 2.56), I got similar results: > > 666 > ok=674 ko=326 > ok=661 ko=339 > ok=791 ko=209 > ok=803 ko=197 > ok=672 ko=328
I still can't reproduce this, not on FreeBSD, not on Arch Linux, not on OpenBSD, not on Cygwin.
> IMHO, as variables can be defined and used mostly anywhere, even using > any kind of sort, you can't assume any order between variables > declarations and uses, so interpolation cannot be done in a single > pass (except when you only allow a single layer of variables, ie bar = > $xyz, foo=$bar not allowed).
I don't understand this sentence. ---- However, I misinterpreted the issue anyway. It doesn't have anything to do with randomness, since Config::General::Interpolate doesn't use random things at all. Sorry for the confusion. So. I've got no idea :(
Download cgsortresults
application/octet-stream 1.2k

Message body not shown because it is not plain text.

Nope. Variable parsing of CONFIG FILES happens in the precise order as they are defined in the config file: Config::General::_parse() => Config::General::_parse_value() => Config::General::Interpolated::_interpolate() Variables are not parsed AFTER parsing is done, but line by line. Take a look: #!/usr/bin/perl use lib qw(blib/lib); use Config::General; use Data::Dumper; my $file = shift || die; my $c = Config::General->new( -ConfigFile => $file, -MergeDuplicateOptions => 1, -MergeDuplicateBlocks => 1, -AllowMultiOptions => 1, -AutoTrue => 1, -InterPolateVars => 1, -SplitPolicy => 'equalsign', -UTF8 => 1 ); my %h = $c->getall(); print Dumper(\%h); Config: a = 1 b = $a c = $b d = $c e = $d f = $e g = $f h = $g i = $h Output: nubic: % perl t.pl t.cfg $VAR1 = { 'b' => 1, 'c' => 1, 'h' => 1, 'e' => 1, 'a' => 1, 'f' => 1, 'i' => 1, 'd' => 1, 'g' => 1 }; This hasn't anything to do with hash ordering, only with occurrence in the input file. Of course it's a completely different story with default hashes. Since hashes in perl are by definition unordered (which is the "random" thing mentioned in the docs), keys inside default hashes are put onto the variable stack by chance. This is, however, no problem at all, unless the config hash ITSELF already uses one of the variables it defines. If it doesn't (and it shouldn't), then order of keys in a default config doesn't matter as well. And as I already said, I still cannot reproduce your results with stock Config::General on any platform at hand. So, there's no problem from my point of view (yet - since I'm still not 100% sure if I understood you correctly!). On thing though: if you need ordered hashes for one or another reason, this kind of stuff is already supported. Refer to the documentation of the -Tie flag, which can be used to tie a sorting hash submodule onto hashes, e.g. by using Tie::IxHash. If you tie your default config as well, everything is ordered. No modification to the code would be required. And - while I'm at it - there are also plugin hooks available for this kind of stuff :) best, Tom
Subject: Re: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work
Date: Tue, 18 Oct 2016 13:41:24 +0000
To: "bug-Config-General [...] rt.cpan.org" <bug-Config-General [...] rt.cpan.org>
From: JusTiCe 8 <justice__8 [...] hotmail.com>
All of this is nice but a bit irrelevant as the issue occurred when I use variables AND default config as the first samples given in this issue report. Show quoted text
> Of course it's a completely different story with default hashes. Since hashes in perl are by definition unordered (which is the "random" thing mentioned in the docs), keys inside default hashes are put onto the variable stack by chance.
The DefaultConfig description is: Show quoted text
> This can be a hash reference or a simple scalar (string) of a config. This causes the module to preset the resulting config hash with the given values, which allows you to set default values for particular config options directly.
Which doesn't tell me not to do so. So, IMHO, you could either choose to dodge the issue by declaring it not handled by Config::General explicitly in the documentation (something like "providing default configuration which can contains variable values is currently forbidden/unmanaged and have to be avoided") or fix it. It's is a bit strange you cannot get the issue (also trying to download the configuration file lead to a BAD REQUEST HTTP error :/ the bugtracker looks like to don't like dot file so much, which may explain why) config file is: %<--- # Config::General variables interpolation issue configuration myvar = something /*item = $myvar*/ <blockone> foo = notbar var = $myvar /* $item */ another = novar /* $myvar */ </blockone> %<------ (as the default contains only "item", "var" and "foo", it looks like Config::General::Interpolated::_interpolate encounter blockone/var BEFORE reading "myvar" value from loaded file) If you haven't done it yet, could you try with the file content above please ? Show quoted text
________________________________ From: T. Linden via RT <bug-Config-General@rt.cpan.org> Sent: Monday, October 17, 2016 11:55 PM To: justice__8@hotmail.com Subject: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work <URL: https://rt.cpan.org/Ticket/Display.html?id=118383 > Nope. Variable parsing of CONFIG FILES happens in the precise order as they are defined in the config file: Config::General::_parse() => Config::General::_parse_value() => Config::General::Interpolated::_interpolate() Variables are not parsed AFTER parsing is done, but line by line. Take a look: #!/usr/bin/perl use lib qw(blib/lib); use Config::General; use Data::Dumper; my $file = shift || die; my $c = Config::General->new( -ConfigFile => $file, -MergeDuplicateOptions => 1, -MergeDuplicateBlocks => 1, -AllowMultiOptions => 1, -AutoTrue => 1, -InterPolateVars => 1, -SplitPolicy => 'equalsign', -UTF8 => 1 ); my %h = $c->getall(); print Dumper(\%h); Config: a = 1 b = $a c = $b d = $c e = $d f = $e g = $f h = $g i = $h Output: nubic: % perl t.pl t.cfg $VAR1 = { 'b' => 1, 'c' => 1, 'h' => 1, 'e' => 1, 'a' => 1, 'f' => 1, 'i' => 1, 'd' => 1, 'g' => 1 }; This hasn't anything to do with hash ordering, only with occurrence in the input file. Of course it's a completely different story with default hashes. Since hashes in perl are by definition unordered (which is the "random" thing mentioned in the docs), keys inside default hashes are put onto the variable stack by chance. This is, however, no problem at all, unless the config hash ITSELF already uses one of the variables it defines. If it doesn't (and it shouldn't), then order of keys in a default config doesn't matter as well. And as I already said, I still cannot reproduce your results with stock Config::General on any platform at hand. So, there's no problem from my point of view (yet - since I'm still not 100% sure if I understood you correctly!). On thing though: if you need ordered hashes for one or another reason, this kind of stuff is already supported. Refer to the documentation of the -Tie flag, which can be used to tie a sorting hash submodule onto hashes, e.g. by using Tie::IxHash. If you tie your default config as well, everything is ordered. No modification to the code would be required. And - while I'm at it - there are also plugin hooks available for this kind of stuff :) best, Tom
On Tue Oct 18 09:41:54 2016, justice__8@hotmail.com wrote: Show quoted text
> All of this is nice but a bit irrelevant as the issue occurred when I > use variables AND default config as the first samples given in this > issue report.
Great, so I don't need to explain anything anymore. Show quoted text
> The DefaultConfig description is: >
> > This can be a hash reference or a simple scalar (string) of a config. > > This causes the module to preset the resulting config hash with the > > given values, which allows you to set default values for particular > > config options directly.
> > Which doesn't tell me not to do so.
Which is not what I said. Show quoted text
> So, IMHO, you could either choose to dodge the issue[..]
If I wanted to "dogge" the issue, I had not responded to it in the first place. Show quoted text
> It's is a bit strange you cannot get the issue
No, it behaves as I expect it to behave. Show quoted text
> If you haven't done it yet, could you try with the file content above > please ?
I already did that, but I did it again: ok=1000 ko=0 It didn't fail, as there's nothing to interpolate in the first place. Just to recount, this is the default config I used (according to your posts): my $config = { item => 1, blockone => { var => 'something', foo => 'bar', }, }; And this is the config with comments removed: myvar = something <blockone> foo = notbar var = $myvar another = novar </blockone> From this, Config:General came up with this: { 'item' => 1, 'myvar' => 'something', 'blockone' => { 'another' => 'novar', 'var' => 'something', 'foo' => 'notbar' } }; Looks legit. If you want me to fix the bug (if there is one), then you have to provide a reproducible test case. Also, you did not respond to the rest of my last comment: did you test it with -Tie and what was the result? - Tom
Subject: Re: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work
Date: Mon, 24 Oct 2016 08:25:15 +0000
To: "bug-Config-General [...] rt.cpan.org" <bug-Config-General [...] rt.cpan.org>
From: JusTiCe 8 <justice__8 [...] hotmail.com>
Show quoted text
> If I wanted to "dogge" the issue, I had not responded to it in the first place.
I just consider any possible case, doesn't mean to offend you in any way, sorry. Show quoted text
> if you want me to fix the bug (if there is one), then you have to provide a reproducible test case.
I did. Unfortunately, the issue I got looks like to be caused by something I don't know which is not equal on our respective setups. Perl offer a way to choose hash function at build time (http://onionstand.blogspot.fr/2012/12/are-you-relying-on-hash-keys-being.html) and probably the version we use were not built using the same config, I don't know. The fact that the exact same code behave in two different ways on two different setups just confuses me. Show quoted text
> Also, you did not respond to the rest of my last comment: did you test it with -Tie and what was the result?
I took time to work on this and I get this (change in bold lead to 1000 ok/0 ko consistently): use strict; use Config::General; use Data::Dumper; use Tie::IxHash; my $config = { ... }; tie %$config, "Tie::IxHash"; my ($oks, $errors) = (0, 0); for (my $count = 0; $count < 1000; $count++) { eval { my $cgobject = Config::General->new ( -ConfigFile => '.configgen-issue.cfg', ... -UTF8 => 1, -Tie => 'Tie::IxHash', ); my %finalconfig = $cgobject->getall (); ... (everything else is identical) So, using Tie::IxHash fix the issue. Thanks for your help on this. Regards. Show quoted text
________________________________ From: T. Linden via RT <bug-Config-General@rt.cpan.org> Sent: Tuesday, October 18, 2016 4:32 PM To: justice__8@hotmail.com Subject: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work <URL: https://rt.cpan.org/Ticket/Display.html?id=118383 > On Tue Oct 18 09:41:54 2016, justice__8@hotmail.com wrote:
> All of this is nice but a bit irrelevant as the issue occurred when I > use variables AND default config as the first samples given in this > issue report.
Great, so I don't need to explain anything anymore.
> The DefaultConfig description is: >
> > This can be a hash reference or a simple scalar (string) of a config. > > This causes the module to preset the resulting config hash with the > > given values, which allows you to set default values for particular > > config options directly.
> > Which doesn't tell me not to do so.
Which is not what I said.
> So, IMHO, you could either choose to dodge the issue[..]
If I wanted to "dogge" the issue, I had not responded to it in the first place.
> It's is a bit strange you cannot get the issue
No, it behaves as I expect it to behave.
> If you haven't done it yet, could you try with the file content above > please ?
I already did that, but I did it again: ok=1000 ko=0 It didn't fail, as there's nothing to interpolate in the first place. Just to recount, this is the default config I used (according to your posts): my $config = { item => 1, blockone => { var => 'something', foo => 'bar', }, }; And this is the config with comments removed: myvar = something <blockone> foo = notbar var = $myvar another = novar </blockone> From this, Config:General came up with this: { 'item' => 1, 'myvar' => 'something', 'blockone' => { 'another' => 'novar', 'var' => 'something', 'foo' => 'notbar' } }; Looks legit. If you want me to fix the bug (if there is one), then you have to provide a reproducible test case. Also, you did not respond to the rest of my last comment: did you test it with -Tie and what was the result? - Tom
Hi, Show quoted text
> Perl offer a way to choose hash function at build time > (http://onionstand.blogspot.fr/2012/12/are-you-relying-on-hash-keys- > being.html)
I don't think this applies to Config::General, since it doesn't rely on key order. Sometimes the user might rely on key order, but not the module itself, that is. Show quoted text
> The fact that the exact same code behave in > two different ways on two different setups just confuses me.
This, indeed, confuses me as well. Really strange. Show quoted text
> So, using Tie::IxHash fix the issue.
Well, so what do you think - close the issue? I'm undecided. best regards, Tom
Subject: Re: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work
Date: Tue, 25 Oct 2016 11:17:55 +0000
To: "bug-Config-General [...] rt.cpan.org" <bug-Config-General [...] rt.cpan.org>
From: JusTiCe 8 <justice__8 [...] hotmail.com>
Show quoted text
> This, indeed, confuses me as well. Really strange.
Have you any idea to move forward on this ? One step could be to get specifics of the systems you have tested on (perl compilation options, soft & hard) as for now and AFAIK, the only difference is us (I have always encounter the issue, you always don't, I may be cursed) Show quoted text
> Well, so what do you think - close the issue? I'm undecided.
me too :) Perhaps you could put it on "stall" as it quite the case. <http://aka.ms/weboutlook> Show quoted text
________________________________ From: T. Linden via RT <bug-Config-General@rt.cpan.org> Sent: Monday, October 24, 2016 10:39 AM To: justice__8@hotmail.com Subject: [rt.cpan.org #118383] Variables interpolation sometimes doesn't work <URL: https://rt.cpan.org/Ticket/Display.html?id=118383 > Hi,
> Perl offer a way to choose hash function at build time > (http://onionstand.blogspot.fr/2012/12/are-you-relying-on-hash-keys- > being.html)
I don't think this applies to Config::General, since it doesn't rely on key order. Sometimes the user might rely on key order, but not the module itself, that is.
> The fact that the exact same code behave in > two different ways on two different setups just confuses me.
This, indeed, confuses me as well. Really strange.
> So, using Tie::IxHash fix the issue.
Well, so what do you think - close the issue? I'm undecided. best regards, Tom