Bug #100658 for Net-FullAuto: documentation lies

Tue Dec 02 05:11:20 2014 tlinden [...] cpan.org - Ticket created

Subject:

documentation lies

I compared the code with what the documentation states. It doesn't correspond. According to the documentation the module is about automating tasks across servers using ssh and stuff like this. In reality the module does, among other things: - requests root permissions here and there (see RT#97421) - installs and removes software (local and remote) like apache, mysql and so forth - maintains amazon ec2 and aws instances - installs and removes perl modules - creates and deletes files and directories in remote and local directories - calls tar, transfers tarballs, unpacks and removes them on various occurences in the code - creates/modifies crontab entries - lot of more stuff I didn't have the time even trying to understand While the documentation talks about security, the module nevertheless stores passwords to berkeley db files. Yes, those are encrypted, but what about the encryption key? It's stored elsewhere, since this thing's main purpose is to run from cron. Obviously the author hasn't ever heard of ssh keys. A couple of notes about the code: it's a bloated mess in one 1.4 mb file, FA_Core.pm. There are 174 subs, many of them are duplicates (e.g. there are 4 subs "cwd"). There are 36 packages within FA_Core.pm, most of them declared inside subs. About 20-30% of the code is redundant, i.e. duplicated all over the module. The file is soo large, that it is impossible to understand for an outsider. It uses hundrets of global variables for everything. While it uses lots of external modules it also does many things on its own (in weird ways, e.g. handling of times and dates), eg: %hourconv=('12:00am'=>0,' 1:00am'=>1,' 2:00am'=>2,' 3:00am'=>3, [..]; [..] $hourstring=$hourconv{unpack('x6 a*',$output->[4]->[0])}; Also there are hardcoded mime-types, path's, programs, etc. The module should - if any - be rewritten from scratch. In its current state it can't be trusted.

Thu Jan 08 12:49:05 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

Subject:

documentation outdated and inaccurate, and other issues

On Tue Dec 02 05:11:20 2014, TLINDEN wrote: Show quoted text

> I compared the code with what the documentation states. It doesn't > correspond. According to the documentation the module is about > automating tasks across servers using ssh and stuff like this. > > In reality the module does, among other things: > > - requests root permissions here and there (see RT#97421) > - installs and removes software (local and remote) like apache, mysql > and so forth > - maintains amazon ec2 and aws instances > - installs and removes perl modules > - creates and deletes files and directories in remote and local > directories > - calls tar, transfers tarballs, unpacks and removes them on various > occurences in the code > - creates/modifies crontab entries > - lot of more stuff I didn't have the time even trying to understand > > While the documentation talks about security, the module nevertheless > stores passwords to berkeley db files. Yes, those are encrypted, but > what about the encryption key? It's stored elsewhere, since this > thing's main purpose is to run from cron. Obviously the author hasn't > ever heard of ssh keys. > > A couple of notes about the code: it's a bloated mess in one 1.4 mb > file, FA_Core.pm. There are 174 subs, many of them are duplicates > (e.g. there are 4 subs "cwd"). There are 36 packages within > FA_Core.pm, most of them declared inside subs. About 20-30% of the > code is redundant, i.e. duplicated all over the module. The file is > soo large, that it is impossible to understand for an outsider. It > uses hundrets of global variables for everything. While it uses lots > of external modules it also does many things on its own (in weird > ways, e.g. handling of times and dates), eg: > > %hourconv=('12:00am'=>0,' 1:00am'=>1,' 2:00am'=>2,' 3:00am'=>3, [..]; > [..] > $hourstring=$hourconv{unpack('x6 a*',$output->[4]->[0])}; > > Also there are hardcoded mime-types, path's, programs, etc. > > The module should - if any - be rewritten from scratch. In its current > state it can't be trusted.

This is a 15 year old POC project. There are reasons for each and every one of your concerns. Some of the reasons are good, some not so good. For instance, the custom date logic was used because it performed much better than any date manipulation modules available at the time. Today, almost certainly a date module can replace most if not all of it. But that's a big task, and as you have witnessed, there are no shortage of "big" tasks this project has left in it. Basically, this monster didn't even "work" until about 4 years ago, and I have been engaged in trying to "package" this thing into a usable utility ever since. There has been a LOT of cleanup already (believe it or not!) and much more is planned. Show quoted text

> The module should - if any - be rewritten from scratch.

This thing actually WORKS and I will put a YouTube video together shortly demonstrating what it can do. It definitely needs re-factoring - but you're clearly suggesting that it be "scrapped", without any true or comprehensive understanding of what this thing is, and what it does. You probably spent 20 minutes trying to make sense out of a 15 year project. There is nothing wrong with that, but I would not expect anyone to be able to comprehend what this is and does by simply doing a quick survey of the source code. This project has had thousands and thousands of development hours patiently poured into it to achieve a very specific goal. That goal "programmatic automation over a PERSISTENT ssh connection" has been achieved. This monster works and its FAST. And it "will" be easy to install and use. I don't know if it will ever be easy to "understand" at a core code level however - no matter how much documentation I produce (and yes, so far, I have produced precious little, and most of that is horribly outdated). I am working to improve EVERYTHING you've outlined, but this is battleship, and it can't be turned quickly. Again, it didn't even "work" until about 4 years ago, and along the way, there was no point in cleaning up code that didn't even work. Now that it works, I've got a titanic mess on my hands - admittedly. But it is a very valuable and ground-breaking, even revolutionary "mess". This *will* be a game changer. It is now, actually, as I have demos built for Amazon that show all of it's impressive capabilities. Please communicate just what out of "all" your concerns you view as the "most" pressing and I will attempt to make some short term headway on it. Show quoted text

> In its current state it can't be trusted.

Yes - I acknowledge that. However it can and will be updated to earn this "trust". It will take time, and it won't happen overnight. But it WILL happen.

Thu Jan 08 12:49:05 2015 The RT System itself - Status changed from 'new' to 'open'

Tue Feb 10 08:17:48 2015 tlinden [...] cpan.org - Correspondence added

On Thu Jan 08 12:49:05 2015, REEDFISH wrote: Show quoted text

> For instance, the custom date logic was used because it > performed much better than any date manipulation modules available at > the time.

DateTime exists since 2003. Show quoted text

> Today, almost certainly a date module can replace most if > not all of it. But that's a big task, and as you have witnessed, there > are no shortage of "big" tasks this project has left in it.

Indeed, but because of your coding style. Show quoted text

> Basically, this monster didn't even "work" until about 4 years ago,

Because it IS a monster. Show quoted text

> There has been a LOT of cleanup already

You're kidding me, aren't you? Show quoted text

> This thing actually WORKS

I didn't criticize if or how it works. The issue at hand is that the documentation mentions about 5% of what the module does and even that 5% are not even true all the time; and that the code does security relevant/dangerous things wihout any notice anywhere. Show quoted text

> and I will put a YouTube video together > shortly demonstrating what it can do.

Don't. POD is for perl module documentation, not Youtube. Show quoted text

> - but you're clearly suggesting that it be "scrapped", without any > true or comprehensive understanding of what this thing is, and what it > does.

If it is not possible to comprehend the purpose and inner workings of some code from looking at it, than it is bad code by definition. So please don't blame me when I don't understand it. Show quoted text

> You probably spent 20 minutes trying to make sense out of a 15 year > project. There is nothing wrong with that, but I would not expect > anyone to be able to comprehend what this is and does by simply doing > a quick survey of the source code.

No one would waste her time by doing this if you only had a propper documentation. Show quoted text

> This project has had thousands and > thousands of development hours patiently poured into it to achieve a > very specific goal. That goal "programmatic automation over a > PERSISTENT ssh connection" has been achieved. This monster works and > its FAST. And it "will" be easy to install and use.

PR brabble isn't required in a RT reply, Thanks. Show quoted text

> I am working to improve EVERYTHING you've outlined, but this is > battleship

There. Show quoted text

> Now that it works, I've got a titanic mess on my hands - admittedly. > But it is a very valuable and ground-breaking, even revolutionary > "mess".

There. Again. Don't you see the discrepancy? Show quoted text

> Please communicate just what out of "all" your concerns you view as > the "most" pressing and I will attempt to make some short term headway > on it.

I already did. Besides, the upmost priority would be to stop spamming CPAN with multiple daily uploads of the module. CPAN is no revision control system. You cannot produce a new release for each single line change. That's crazy. Use git or svn for this and make a new release if a couple of bugs have been fixed or a new feature is completely implemented. Show quoted text

> > In its current state it can't be trusted.

> > Yes - I acknowledge that.

Then, please, act accordingly. - Tom

Tue Feb 10 23:53:16 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

CC:	tlinden [...] cpan.org
Subject:	Re: [rt.cpan.org #100658] documentation lies
Date:	Tue, 10 Feb 2015 23:52:56 -0500
To:	bug-Net-FullAuto [...] rt.cpan.org
From:	brian.kelly [...] fullautosoftware.net

I'm not using CPAN as source control. I'm not "spamming" either - in the sense of trying to "sell" this module in an unorthodox fashion. The multiple uploads are work on the installer. FullAuto has a LOT of dependency modules. Two years ago it took the better part of a full day to get FullAuto and all its dependencies installed. With that kind of effort required to get it running, no one was going to use it. So the last two years was basically spent in solving "that" problem. Now FullAuto can be installed with ALL it's dependencies in less than 15 minutes - often in as little as 8 minutes. I tried to use CPAN as documented for this task, but it proved woefully inadequate. BerkeleyDB is a case in point. The FullAuto install now installs this without intervention on all Linux/Unix and Windows with Cygwin. I've tried using local mirrors, and while that allowed some level of independence from CPAN, there were issues that only appeared when using CPAN itself, especially when installing on Amazon EC2 micro instances. I'm not "comfortable" that I'm clearly irritating you - and yes, I will concede that you are very likely the "voice" of others as well. The number of uploads has slowed down, and will continue to do so. My goal is to get to a release schedule that spans months - not days. 90% of the uploads are a direct consequence of CPAN issues, or issues with other author's modules and how they get installed, and my attempts to work it all out - I am not doing it as an alternative to svn or git. As for documentation, nobody is using this yet, for reasons you have outlined in addition to others. The software that is closest to what FullAuto does is Chef and Puppet. (And yes, I can hear your laughter from here). You won't be laughing for long. Everything you're complaining about is fixable, and it WILL get fixed. What FullAuto does, and how it does it will insure that will happen. Chef requires 10 GIG of hard disk space on every node. Chef requires TWO way communication between nodes and the Chef server. FullAuto requires ZERO disk space on ANY node, and requires only a one-way connection from the FullAuto installation to the node via PERSISTENT SSH. FullAuto runs commands from one script on one host and controls any number of other hosts simultaneously - again via persistent SSH. The FullAuto demo that is nearly complete, installs itself on an Amazon micro server, and then proceeds to use the Amazon API to launch other servers, download Liferay (LDAP is another demo. Hadoop is next), build it from source, install MySQL on yet another host, start it, download and import Liferay's schema, and exchange configuration information among the hosts so in the end a multi-host application infrastructure starts up and is production ready - without ANY manual intervention. FullAuto can stand up any number of servers, install Chef, configure it on every node, and start it - with NO manual intervention. Chef in turn, cannot do the same, because before it can do anything, it has to be installed EVERYWHERE. FullAuto has no such dependency. Even it's own installation is "FULL" - Auto. That is why it's Makefile.PL file is over 7000 lines of code. Many pundits have opined that automation utilities like Chef and Puppet are woefully under-utilized. I AGREE - and it's because both of them are difficult to setup and use. I am trying to remedy that. But the cost of that is insanely complex code. There is always a trade-off. FullAuto dynamically discovers the prompt on remote hosts, dynamically discovers the shell and the location of all key utilities. FullAuto returns stdout, stderr and return codes from each command it runs on the remote hosts. All FullAuto needs to fully control a remote host is a hostname or ip address, login id, and password or key file. Basically, anything anyone can do manually over a SSH/SFTP (telnet or ftp) connection, FullAuto can do programmatically, and with minimal code. FullAuto even cleans up and shuts down everything automatically - even after a CTRL-C interrupt. And all this power runs on a MICRO server. Please point me to anything else that does all this? Something this all-encompassing is going to be seriously complex, and for me personally, having everything in a few files is easier for me to cope with than everything broken out into a gazillion modules. Some of that will change soon, but probably not as much as you would probably advocate. Show quoted text

> I didn't criticize if or how it works. The issue at hand is that the > documentation mentions about 5% of what the module does and even that 5% > are not even true all the time; and that the code does security > relevant/dangerous things wihout any notice anywhere.

CPAN itself uses sudo and has it as one of it's setup options. There's not a "lot" of documentation on it's internal usage (short of studying the code itself), and I haven't noticed any "notice" about when and how it decides to use it once you elect it as an option. So why are you criticizing "me" for doing the same? You can't even use apt-get or yum without sudo. How in the world am I to automate ANYTHING without occasional judicial and carefully invoked super-user access, and if I'm asking the user for permission every time I do use sudo (which might be dozens and dozens of times during a multi-host standup), then we don't have a "FullAuto" process - do we? And by the way, Amazon doesn't use passwords (except for Windows), only key files. So by extension, FullAuto uses only key files when working against Amazon. I want people to seriously use this "someday" and of course they won't without adequate documentation - so that WILL be remedied, and soon. Again, I am not "trying" to irritate you or anyone else. When I have problems with CPAN and third party module installation that I cannot remedy or adequately debug and fix with a local mirror, what else would you have me do? Show quoted text

> Queue: Net-FullAuto > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=100658 > > > On Thu Jan 08 12:49:05 2015, REEDFISH wrote:

>> For instance, the custom date logic was used because it >> performed much better than any date manipulation modules available at >> the time.

> > DateTime exists since 2003. >

>> Today, almost certainly a date module can replace most if >> not all of it. But that's a big task, and as you have witnessed, there >> are no shortage of "big" tasks this project has left in it.

> > Indeed, but because of your coding style. >

>> Basically, this monster didn't even "work" until about 4 years ago,

>> Because it IS a monster

>> There has been a LOT of cleanup already

> > You're kidding me, aren't you? >

>> This thing actually WORKS

> > I didn't criticize if or how it works. The issue at hand is that the > documentation mentions about 5% of what the module does and even that 5% > are not even true all the time; and that the code does security > relevant/dangerous things wihout any notice anywhere. >

>> and I will put a YouTube video together >> shortly demonstrating what it can do.

> > Don't. POD is for perl module documentation, not Youtube. >

>> - but you're clearly suggesting that it be "scrapped", without any >> true or comprehensive understanding of what this thing is, and what it >> does.

> > If it is not possible to comprehend the purpose and inner workings of some > code from looking at it, than it is bad code by definition. So please > don't blame me when I don't understand it. >

>> You probably spent 20 minutes trying to make sense out of a 15 year >> project. There is nothing wrong with that, but I would not expect >> anyone to be able to comprehend what this is and does by simply doing >> a quick survey of the source code.

> > No one would waste her time by doing this if you only had a propper > documentation. >

>> This project has had thousands and >> thousands of development hours patiently poured into it to achieve a >> very specific goal. That goal "programmatic automation over a >> PERSISTENT ssh connection" has been achieved. This monster works and >> its FAST. And it "will" be easy to install and use.

> > PR brabble isn't required in a RT reply, Thanks. >

>> I am working to improve EVERYTHING you've outlined, but this is >> battleship

> > There. >

>> Now that it works, I've got a titanic mess on my hands - admittedly. >> But it is a very valuable and ground-breaking, even revolutionary >> "mess".

> > There. Again. Don't you see the discrepancy? >

>> Please communicate just what out of "all" your concerns you view as >> the "most" pressing and I will attempt to make some short term headway >> on it.

> > I already did. > > Besides, the upmost priority would be to stop spamming CPAN with multiple > daily uploads of the module. CPAN is no revision control system. You > cannot produce a new release for each single line change. That's crazy. > Use git or svn for this and make a new release if a couple of bugs have > been fixed or a new feature is completely implemented. >

>> > In its current state it can't be trusted.

>> >> Yes - I acknowledge that.

> > Then, please, act accordingly. > > > > - Tom > > >

Wed Feb 11 06:57:12 2015 tlinden [...] cpan.org - Correspondence added

On Tue Feb 10 23:53:16 2015, REEDFISH wrote: Show quoted text

> FullAuto has a LOT of dependency modules.

I took a look at your Makefile.PL: WHAT THE HELL ARE YOU DOING?! There are some simple rules for perl module installers: - a perl module doesn't install any other perl module - it doesn't assume root or admin permissions (hint: perl Makefile.PL PREFIX=$HOME/lib; make install) - this one is a NO GO: open(LD,"sudo yum -y install 'openssl-devel'|"); - it declares its dependencies - use at least ExtUtils::MakeMaker - 275k for Makefile.PL is 274k too much Show quoted text

> 90% of the uploads are a direct consequence of CPAN issues, or issues with > other author's modules and how they get installed, and my attempts to work > it all out - I am not doing it as an alternative to svn or git.

That's not true. From version 0.999999999983 to 0.999999999984 only 1 character of code changed: diff -cr Net-FullAuto-0.999999999983/lib/Net/FullAuto/FA_Core.pm Net-FullAuto-0.999999999984/lib/Net/FullAuto/FA_Core.pm *** Net-FullAuto-0.999999999983/lib/Net/FullAuto/FA_Core.pm 2015-02-08 17:20:56.000000000 +0100 --- Net-FullAuto-0.999999999984/lib/Net/FullAuto/FA_Core.pm 2015-02-09 00:34:16.000000000 +0100 *************** *** 14820,14826 **** my $credentials_csv_path='.';my $fa_tag=0; if (-r "/tmp/fa_aws_home.txt") { open(RD,"/tmp/fa_aws_home.txt"); ! while (my $line=~<RD>) { $credentials_csv_path=$line if $line=~/home/; $fa_tag=$line if $line=~/TagFA/; $fa_tag=~s/^.*=(\d)\s*$/$1/ if $fa_tag; --- 14820,14826 ---- my $credentials_csv_path='.';my $fa_tag=0; if (-r "/tmp/fa_aws_home.txt") { open(RD,"/tmp/fa_aws_home.txt"); ! while (my $line=<RD>) { $credentials_csv_path=$line if $line=~/home/; $fa_tag=$line if $line=~/TagFA/; $fa_tag=~s/^.*=(\d)\s*$/$1/ if $fa_tag; You removed '~' and made a new release for this, which is what you do every day. I'll not comment the rest of your response, sorry. - Tom

Wed Feb 11 07:37:17 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

Subject:	Re: [rt.cpan.org #100658] documentation lies
Date:	Wed, 11 Feb 2015 07:36:47 -0500
To:	bug-Net-FullAuto [...] rt.cpan.org
From:	brian.kelly [...] fullautosoftware.net

So I go back to taking an entire day for setup? Seriously? I'd happily let ExtUils::MakeMaker do everything - if it could. It's there - it's in the mix. Show quoted text

>> WHAT THE HELL ARE YOU DOING?!

I'd be more than happy to explain everything, every line of code, if you were "actually" interested - which you're clearly not. You've already tried, deliberated and condemned. Am I wrong? Show quoted text

> Queue: Net-FullAuto > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=100658 > > > On Tue Feb 10 23:53:16 2015, REEDFISH wrote:

>> FullAuto has a LOT of dependency modules.

> > I took a look at your Makefile.PL: > > WHAT THE HELL ARE YOU DOING?! > > There are some simple rules for perl module installers: > > - a perl module doesn't install any other perl module > - it doesn't assume root or admin permissions > (hint: perl Makefile.PL PREFIX=$HOME/lib; make install) > - this one is a NO GO: > open(LD,"sudo yum -y install 'openssl-devel'|"); > - it declares its dependencies > - use at least ExtUtils::MakeMaker > - 275k for Makefile.PL is 274k too much >

>> 90% of the uploads are a direct consequence of CPAN issues, or issues >> with >> other author's modules and how they get installed, and my attempts to >> work >> it all out - I am not doing it as an alternative to svn or git.

> > That's not true. From version 0.999999999983 to 0.999999999984 only 1 > character of code changed: > > diff -cr Net-FullAuto-0.999999999983/lib/Net/FullAuto/FA_Core.pm > Net-FullAuto-0.999999999984/lib/Net/FullAuto/FA_Core.pm > *** Net-FullAuto-0.999999999983/lib/Net/FullAuto/FA_Core.pm 2015-02-08 > 17:20:56.000000000 +0100 > --- Net-FullAuto-0.999999999984/lib/Net/FullAuto/FA_Core.pm 2015-02-09 > 00:34:16.000000000 +0100 > *************** > *** 14820,14826 **** > my $credentials_csv_path='.';my $fa_tag=0; > if (-r "/tmp/fa_aws_home.txt") { > open(RD,"/tmp/fa_aws_home.txt"); > ! while (my $line=~<RD>) { > $credentials_csv_path=$line if $line=~/home/; > $fa_tag=$line if $line=~/TagFA/; > $fa_tag=~s/^.*=(\d)\s*$/$1/ if $fa_tag; > --- 14820,14826 ---- > my $credentials_csv_path='.';my $fa_tag=0; > if (-r "/tmp/fa_aws_home.txt") { > open(RD,"/tmp/fa_aws_home.txt"); > ! while (my $line=<RD>) { > $credentials_csv_path=$line if $line=~/home/; > $fa_tag=$line if $line=~/TagFA/; > $fa_tag=~s/^.*=(\d)\s*$/$1/ if $fa_tag; > > You removed '~' and made a new release for this, which is what you do > every day. > > I'll not comment the rest of your response, sorry. > > > - Tom >

Fri Feb 27 17:51:57 2015 ether [...] cpan.org - Correspondence added

RT-Send-CC:

Brian.Kelly [...] fullautosoftware.net

On 2015-02-10 20:53:16, REEDFISH wrote: Show quoted text

> I've tried using local mirrors, and while that allowed some level of > independence from CPAN, there were issues that only appeared when using > CPAN itself, especially when installing on Amazon EC2 micro instances. > > 90% of the uploads are a direct consequence of CPAN issues, or issues with > other author's modules and how they get installed, and my attempts to work > it all out - I am not doing it as an alternative to svn or git.

Hi, I'm a member of the Perl Toolchain Gang, the working group tasked with maintaining and developing the core toolchain modules. Can you elaborate on your "cpan issues" here that necessitate new uploads? What are you trying to do, what do you expect to happen, and what is happening instead? Have you reported any of these issues?

Sat Feb 28 13:57:09 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

CC:	ether [...] cpan.org, ej_zg [...] hotmail.com
Subject:	Re: [rt.cpan.org #100658] documentation lies
Date:	Sat, 28 Feb 2015 13:56:45 -0500
To:	bug-Net-FullAuto [...] rt.cpan.org
From:	brian.kelly [...] fullautosoftware.net

Hi Karen and Ed - thanks for your interest. I'd like to correct that statement. It's not that I'm having issues with "CPAN" per se, but rather with modules that I use as dependencies for Net::FullAuto. One of the biggest problems I've had making FullAuto "accessible" to anyone who wants to try or use it, was the lengthy, and very complicated installation and configuration. FullAuto uses a whopping 137 (not including FullAuto itself) dependency modules from CPAN. Most of those are of course dependencies of dependencies. Send::Email - a module included in FullAuto's dependency chain, has a huge number of dependencies all on it's own. So does Moose of course. As good as the cpan shell and installer are, I could never get it to correctly manage this massive dependency chain. Inevitably, the installation would fail because of a missing dependency in one of the long sub-chains of dependencies. Not necessarily the fault of CPAN itself either, because module authors themselves are of course responsible for making sure their stuff plays well with others. (And yes, I am not unaware of the obvious hypocrisy of that statement). One noted example is Module::Build "recommends" Pod::Readme - but Pod::Readme requires Module::Build to be installed! A classic circular dependency. Now of course, this is precisely why it's "recommended" and not "required" - but I still spent upwards of half a day some time back fighting with this because I try to supply recommended modules as much as possible. The answer was of course to install Pod::Readme AFTER a full install of Module::Build, but that became obvious only in hindsight. Today - I don't include Pod::Readme, because it turns out it has a dependency chain of it's own that I decided was too much baggage for a module that was only "recommended". There is clearly a trend and philosophy to break out more and more functionality into separate modules, and while there are clearly benefits to this trend, the cost is increased complexity in the dependency tree of modules like mine. With a dependency tree numbering 137 and growing, hardly a day goes by that someone isn't updating one of more of these modules. That alone has created massive problems for me that the generic cpan module is simply not robust enough to cope with. Inevitably some module in the dependency chain will require a newer version of a module than is available in the local Perl installation. On some occasions, the updated module will be installed into the SITE branch, but the older module is in the VENDOR branch - so now you have two conflicting modules on the system and because VENDOR is usually the parent in the @INC array, the installing module finds the older one and breaks. Even if I delete the offending older module, I have to re-run Makefile.PL from the top in order to pick up the new correct module - doing that with an exec. Then there's the problem of some module authors asking for user input. I'm trying to make FullAuto "consumer" available and not just to Perl programmers familiar with cpan and Perl idiosyncrasies. One of the JSON modules does this, and a "consumer" (again, someone not familiar with Perl and/or CPAN) is not going to know how to answer such a query. Not to mention, I want a seamless, non-interactive install for the entire dependency chain and this one module alone breaks that. So I needed a way to feed arguments to Makefile.PL that cpan simply doesn't provide a facility for (that I am aware of - I could be wrong of course). So, I had to build my own. BerekelyDB is another classic example of a module needing arguments fed to it via either arguments to Makefile.PL or via environment variables. Then there's Win32::API which still doesn't work without a patch on 64 bit Windows. The author has yet to fix this (no foul, everyone is volunteering, and no one is "obligated"), but I can't wait months of even years. So I need a way to dynamically patch such modules myself during installation. Again something the generic cpan does not have a facility for (again, that I am aware of). I know others have opened tickets on the issue with the author - which only fueled the need for a fix I could introduce myself. Again, I can't afford to wait. And finally, cpan ran out of memory on some Amazon EC2 micro machine instances when trying to install modules such as Email::Sender with large dependency trees. The only way I could work around this was to install all it's dependencies myself before I installed Email::Sender - and other modules. FullAuto works extremely well on Amazon micro instances, and since that is going to be one of its big advantages, (versus competitors like Chef and Puppet), having memory and installation issues on these hosts was of course "not acceptable". There are yet more reasons I had to deviate from protocol, but the point is there are "reasons" for every line of code in my Makefile.PL. I am chasing a 15 year goal - and that goal is now in reach. I had to make FullAuto ACCESSIBLE to users, because no one is going to spend a day or more on a complex installation with hundreds of opportunities for human error to complicate things even more. There was simply no other way for me to proceed. (Again, that I was or am aware of - I am open to any and all suggestions). But all that said, I would have LOVED and would LOVE to have access to a CPAN that can deliver a seamless installation of a complex architecture - and I will try and assist in any way I can to help make that happen. More than likely though, that will be accomplished by others someday - and perhaps with the help of paid FullAuto contributors even. I have done my best to turn off CPAN Testing - as it has come to my attention that the FullAuto installation was causing problems. For that I apologize, it is not my desire to cause issues for anyone, especially volunteers who are being extremely generous with their time. If there is something else I need to do, please let me know. Ed - I am not using CPAN for source control, though I acknowledge that it can appear that way to outsiders. I have my own local svn setup I use. I do plan to set up GIT on Sourceforge, but that task is on a LONG list of to-dos. The frequent uploads are about corrections made to the entire FullAuto installation chain of events that cpan is an integral part of. Most of this type of work is finished, and the frequency of uploads will decrease dramatically. I am including you on this email reply to Karen, because this reply contains details about why there has been so much activity. Again, thanks for your interest. Brian Kelly Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=100658 > > > On 2015-02-10 20:53:16, REEDFISH wrote:

>> I've tried using local mirrors, and while that allowed some level of

independence from CPAN, there were issues that only appeared when using CPAN itself, especially when installing on Amazon EC2 micro instances. 90% of the uploads are a direct consequence of CPAN issues, or issues with Show quoted text

>> other author's modules and how they get installed, and my attempts to

work Show quoted text

>> it all out - I am not doing it as an alternative to svn or git.

> > Hi, I'm a member of the Perl Toolchain Gang, the working group tasked

with Show quoted text

> maintaining and developing the core toolchain modules. Can you

elaborate Show quoted text

> on > your "cpan issues" here that necessitate new uploads? What are you

trying Show quoted text

> to > do, what do you expect to happen, and what is happening instead? Have

you Show quoted text

> reported any of these issues? > >

Sun Mar 01 05:11:50 2015 KENTNL [...] cpan.org - Correspondence added

On 2015-03-01 07:57:09, REEDFISH wrote:
> FullAuto uses a whopping 137 (not including FullAuto
> itself) dependency modules from CPAN. Most of those are of course
> dependencies of dependencies. Send::Email - a module included in
> FullAuto's dependency chain, has a huge number of dependencies all on it's
> own. So does Moose of course. As good as the cpan shell and installer are,
> I could never get it to correctly manage this massive dependency chain.
> Inevitably, the installation would fail because of a missing dependency in
> one of the long sub-chains of dependencies. Not necessarily the fault of
> CPAN itself either, because module authors themselves are of course
> responsible for making sure their stuff plays well with others.

There are some practical solutions for these problems outside stomping on files that aren't yours.

For instance, some applications use "Fatpacking", an approach where cpan modules are essentially "Inlined" as a step in the release process.

Another approach is "bundling", where you store the trees of dependencies you *can* safely install directly in somewhere like bundled/ , and then have code in Makefile.PL that copies those files into a prefix.

For instance, say you wanted Class::Tiny to be "bundled" for some reason.

You *could* do this by storing it in

bundled/Class/Tiny.pm

And then during Makefile.PL, copy them to the build target as:

lib/Net/FullAuto/bundled/Class/Tiny.pm

And then load that path into @INC prior to loading the dependencies from it.

( like http://is.gd/l9cgzI  )

And then you can disable the "install bundled" logic conditionally upon an ENV flag and revert back to "use dependencies",

( like https://metacpan.org/source/BINGOS/ExtUtils-MakeMaker-7.04/Makefile.PL#L49 )

This helps you for people who *want* to use CPAN deps and it certainly helps you better when it comes to vendor deployment ( But the state of your current codebase, just in Makefile.PL is such no vendor would touch it ). Keeping in mind, vendors do not need you to dumb things down for them, they're quite capable of sorting out dependencies =).

But in short, you've been heading on your own in this direction for a long time, and this is actually working against you. Many of your problems and solutions could have reached better results had to come to #toolchain on irc.perl.org and chatted about them.

And we really do encourage that so we can work out what your objective problems are one at a time, and find objective solutions to them that work for more people, and work in the way people who generally consume CPAN expect.

Think of it as free help that will make your code better, _and_ attract more users ;D

Mon Mar 02 08:18:21 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

Subject:	Re: [rt.cpan.org #100658] documentation lies
Date:	Mon, 2 Mar 2015 08:18:02 -0500
To:	bug-Net-FullAuto [...] rt.cpan.org
From:	brian.kelly [...] fullautosoftware.net

Thanks Frederick, This is GOOD advice - some of it I wasn't aware of. I actually can create a fully bundled distribution, but your advice would certainly have made things easier. I will study your advice and see what kind of both short and longer term changes I can/need to make bring this "monster" more in harmony with the CPAN ecosystem. Thanks for the response! Brian Kelly Show quoted text

> Queue: Net-FullAuto > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=100658 > > > On 2015-03-01 07:57:09, REEDFISH wrote:

>> FullAuto uses a whopping 137 (not including FullAuto >> itself) dependency modules from CPAN. Most of those are of course >> dependencies of dependencies. Send::Email - a module included in >> FullAuto's dependency chain, has a huge number of dependencies all on >> it's >> own. So does Moose of course. As good as the cpan shell and installer >> are, >> I could never get it to correctly manage this massive dependency chain. >> Inevitably, the installation would fail because of a missing dependency >> in >> one of the long sub-chains of dependencies. Not necessarily the fault of >> CPAN itself either, because module authors themselves are of course >> responsible for making sure their stuff plays well with others.

> > There are some practical solutions for these problems outside stomping on > files > that aren't yours. > > For instance, some applications use "Fatpacking", an approach where cpan > modules are essentially "Inlined" as a step in the release process. > > Another approach is "bundling", where you store the trees of dependencies > you > *can* safely install directly in somewhere like bundled/ , and then have > code > in Makefile.PL that copies those files into a prefix. > > For instance, say you wanted Class::Tiny to be "bundled" for some reason. > > You *could* do this by storing it in > > bundled/Class/Tiny.pm > > And then during Makefile.PL, copy them to the build target as: > > lib/Net/FullAuto/bundled/Class/Tiny.pm > > And then load that path into @INC prior to loading the dependencies from > it. > > ( like http://is.gd/l9cgzI ) > > And then you can disable the "install bundled" logic conditionally upon an > ENV > flag and revert back to "use dependencies", > > ( like > https://metacpan.org/source/BINGOS/ExtUtils-MakeMaker-7.04/Makefile.PL#L49 > ) > > This helps you for people who *want* to use CPAN deps and it certainly > helps > you better when it comes to vendor deployment ( But the state of your > current > codebase, just in Makefile.PL is such no vendor would touch it ). Keeping > in > mind, vendors do not need you to dumb things down for them, they're quite > capable of sorting out dependencies =). > > But in short, you've been heading on your own in this direction for a long > time, and this is actually working against you. Many of your problems and > solutions could have reached better results had to come to #toolchain on > irc.perl.org and chatted about them. > > And we really do encourage that so we can work out what your objective > problems > are one at a time, and find objective solutions to them that work for more > people, and work in the way people who generally consume CPAN expect. > > Think of it as free help that will make your code better, _and_ attract > more > users ;D > >

Mon Mar 02 10:41:33 2015 tlinden [...] cpan.org - Correspondence added

On Sun Mar 01 05:11:50 2015, KENTNL wrote: Show quoted text

> Another approach is "bundling", where you store the trees of > dependencies you *can* safely install directly in somewhere > like bundled/ , and then have code in Makefile.PL that copies > those files into a prefix.

While this is a good advise in general, I don't agree in this particular case. The root cause of the problems the author has, is that his module does just WAY too many things and there are established solutions to most (if not all) of them. It might sometimes be a good idea to implement things differently in order to progress. But I don't see it in this case. It is also difficult to achieve in practise. You need to install more than a hundred modules on lots of different platforms. That's impossible to maintain for one person. Therefore we're using modules in the first place, don't we? So, I'd stay with my first advise: reduce the complexity of the module and/or split it off into separate pieces (one for the actual automation, one for amazon aws maintenance, one for xen or whatnot) and try it first with a bare minimum Makefile.PL (as I already suggested) and then work out where problems might remain. best, Tom

Mon Mar 02 16:30:35 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

RT-Send-CC:

ether [...] cpan.org, ej_zg [...] hotmail.com

Hi Tom, Thanks for the response. A ticket may not be the "best" place for discussion of general principles, but since we're exploring the best method of solving my multiple problems, it might be reasonable for me to share my perspective on what my "problems" are. I can certainly appreciate the concern my module arouses. It is admittedly an "arrogant" module. Arrogant in the sense that (especially in regards to it's installation) the implicit if not always obvious assumption is that FullAuto will essentially "own" the local Perl installation. For ANY module to have such an assumption is anathema for computing going backwards in time. I don't argue that at all. In fact, there IS a built in WARNING screen in the Makefile.PL that highly suggests the use of an entirely separate Perl installation, that indeed, FullAuto can "own". I'll have to review the code path to the WARNING and make sure it still appears for everyone. But what about going FORWARD??? An Amazon EC2 micro server, is about as disposable a commodity if there ever was one. Free Tier for a year, about $120 a year after that, and launch-able and destroy-able in a handful of minutes, is it really "unthinkable" for a powerful multi-host capable module to essentially "assume" it has a box/or Perl installation all to itself? (And that is an exaggeration too, because FullAuto can indeed live happily will all kinds of other modules and scripts - the real issue is the need for updated modules that might break older scripts and other modules. I understand that - Kent's advice is right on target with that need). And since you can now create and destroy whole servers in less time than it takes to use the toilet, is it unreasonable for guys like me to see a need for software installations that can leverage and "keep pace" with that reality, even it means getting a bit creative in the short term? What good is a server that can be launched in 2 minutes, if it then takes 2 days to set it up? And yes, there is YUM and APT and RPM - but what about Windows? FullAuto works fully with Cygwin. Plus with FullAuto, you have infinitely more potential configuration flexibility than you do with any repository based images - whether machine or software. I have just recently uploaded a video to YouTube demonstrating the full power and capability of this module. I invite all who are interested to view it: http://youtu.be/FfC_Ybdq07w I'd be VERY interesting in hearing some of your perspectives on this, as it is VERY germane to the direction the future of this module "should" go in. Do I continue to build it up - or break it down? Can I get the exact same results you will see in the video while following Tom's advice to the extent he advocates? Am I doing "too much"? And if so, why is that a "bad" thing going forward? Show quoted text

>> his module does just WAY too many things and there are >> established solutions to most (if not all) of them

Please watch the video and then please tell me what other "established solutions" do the same thing? (The video is 50 minutes long, you skip around and in just a few minutes get the "gist" of what it does). Thanks, Brian Kelly On Mon Mar 02 10:41:33 2015, TLINDEN wrote: Show quoted text

> On Sun Mar 01 05:11:50 2015, KENTNL wrote:

> > Another approach is "bundling", where you store the trees of > > dependencies you *can* safely install directly in somewhere > > like bundled/ , and then have code in Makefile.PL that copies > > those files into a prefix.

> > While this is a good advise in general, I don't agree in this > particular case. The root cause of the problems the author has, is > that his module does just WAY too many things and there are > established solutions to most (if not all) of them. It might sometimes > be a good idea to implement things differently in order to progress. > But I don't see it in this case. > > It is also difficult to achieve in practise. You need to install more > than a hundred modules on lots of different platforms. That's > impossible to maintain for one person. Therefore we're using modules > in the first place, don't we? > > So, I'd stay with my first advise: reduce the complexity of the module > and/or split it off into separate pieces (one for the actual > automation, one for amazon aws maintenance, one for xen or whatnot) > and try it first with a bare minimum Makefile.PL (as I already > suggested) and then work out where problems might remain. > > > best, > Tom

Mon Mar 02 16:42:34 2015 ETJ [...] cpan.org - Correspondence added

Arrogance is not itself necessarily fatal: all software engineering goes from a thing not being possible, to being possible and done. However, proceeding from badly mistaken assumptions is fatal. This module is founded on the assertion that the only way to do the things sought, is in the way it is being done here. In particular, the idea that the only way to get things done is to "take over" the perl installation is not just badly flawed, but fatally so. That is why you have spent 15 years and are still only "nearly there". If you were prepared to revisit all your assumptions, and discuss them with the Perl Toolchain Gang on irc.perl.org/#irc, I think you would make rapid, significant progress.

Tue Mar 03 03:27:41 2015 tlinden [...] cpan.org - Correspondence added

Brian, so I watched your video, at least parts of it. The problem is this: you're talking about WHAT your software does and how unique it is and so forth. But that's not the point of the discussion here. We're talking about HOW you do things. I understand what you're trying to achieve, but you're not doing it the "perl way". Besides, you say you're not aware of any other technology for such things. Might be, but that doesn't mean that this assumtion is correct as Ed said as well. Especially in professional environments you're not working this way. Just to make it clear let me tell you how I'm doing this at work: I maintain multiple dmz's, each one with 50-100 servers, where each server is a HP blade. All of them run FreeBSD, and on top of it runs vast numbers of jails with services (~1000 services per dmz). Each jail comes with its mounts (via nfs), ip config, firewall, software, config, data, logging and whatnot. Jails can be moved from one server to another with one command and across dmz's. So, if there's a new blade, I just boot it up. I boots a freebsd installer via netboot, installs a preconfigured image, registers itself etc and reboots. At this point the new blade is visible in monitoring, capacity management etc. On our management system I enter "uhd-srv3455 add hjh-btomcat4" and a jail with a tomcat for the customer "hjh" will be moved to the new blade and started up. The whole process from installation to the first service running takes a couple of minutes. So, that's how we do it. Others do it differently, some with more manual work than others, some use commercial tools for this, some use inhouse solutions. There is a huge market for server deployment solutions and there are literally hundreds of systems available. I don't like bashing on your module all the time, but you need to tune your assumptions and preconceptions in order to make anything good of it. best, Tom

Wed Mar 04 08:54:56 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

Subject:	Re: [rt.cpan.org #100658] documentation lies
Date:	Wed, 4 Mar 2015 08:54:31 -0500
To:	bug-Net-FullAuto [...] rt.cpan.org
From:	brian.kelly [...] fullautosoftware.net

Thanks Tom, Thanks for checking out the video. And I can't argue with you, there is a LOT of things that have to be refactored. The reason I've focused so much on the "what" is because I'm actually pursuing investment capital - and there is genuine interest. With some of the money, I would hire a perl consultant or two to focus JUST on the installation problems. A tech writer or two for the documentation and training materials. Even a BerkeleyDB consultant to tune the way I'm using that. Heck - I might even throw some money at the toolchain gang, if they'd be interested in getting their hands "dirty". I intend to make a proper lady out of this thing one way or another. And yes, you're right, there are indeed a LOT of deployment tools. But I like to think that FullAuto is "more" than a mere deployment tool, which is why I've called it a "process automation utility". I can connect to anything SSH "persistently". Even Anisible, which claims to be "agentless" has a dependency on Python being on the remote nodes. FullAuto has no such dependency. As such, it can connect to routers and switches and other devices that do not have Python available. Anisible also does not work on Windows - FullAuto does with Cygwin. You mentioned "pre-configured" images - the demo you saw manipulated a "naked" image. NO pre-configured images to store or have to version. A "pre-configured" image is essentially a hard-coded image (loosely speaking). Instead of a dozen "pre-configured" images for different components, you can have just ONE FullAuto instruction set, that can have infinite configuration change options. But you're right - that is a bit "off topic". Thanks for sharing your experience - very interesting. Show quoted text

> Queue: Net-FullAuto > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=100658 > > > Brian, > > so I watched your video, at least parts of it. The problem is this: you're > talking about WHAT your software does and how unique it is and so forth. > But that's not the point of the discussion here. We're talking about HOW > you do things. I understand what you're trying to achieve, but you're not > doing it the "perl way". > > Besides, you say you're not aware of any other technology for such things. > Might be, but that doesn't mean that this assumtion is correct as Ed said > as well. Especially in professional environments you're not working this > way. Just to make it clear let me tell you how I'm doing this at work: > > I maintain multiple dmz's, each one with 50-100 servers, where each server > is a HP blade. All of them run FreeBSD, and on top of it runs vast numbers > of jails with services (~1000 services per dmz). Each jail comes with its > mounts (via nfs), ip config, firewall, software, config, data, logging and > whatnot. Jails can be moved from one server to another with one command > and across dmz's. > > So, if there's a new blade, I just boot it up. I boots a freebsd installer > via netboot, installs a preconfigured image, registers itself etc and > reboots. At this point the new blade is visible in monitoring, capacity > management etc. On our management system I enter "uhd-srv3455 add > hjh-btomcat4" and a jail with a tomcat for the customer "hjh" will be > moved to the new blade and started up. The whole process from installation > to the first service running takes a couple of minutes. > > So, that's how we do it. Others do it differently, some with more manual > work than others, some use commercial tools for this, some use inhouse > solutions. There is a huge market for server deployment solutions and > there are literally hundreds of systems available. > > I don't like bashing on your module all the time, but you need to tune > your assumptions and preconceptions in order to make anything good of it. > > > > best, > Tom >

Wed Mar 04 09:09:49 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

On Wed Mar 04 08:54:56 2015, REEDFISH wrote: Show quoted text

> Thanks Tom, > > Thanks for checking out the video.

The video link has changed: http://youtu.be/gRwa1QoOS7M Thanks, Brian Kelly

Sun Mar 08 11:08:50 2015 KENTNL [...] cpan.org - Correspondence added

My intention with showing the sample solution I did was to expose you to the sorts of solutions and knowledge that can be gained from the CPAN community, specifically toolchain operators, who have combined experience of well over 100 years =).

The suggestion was not so much to take the suggestion gratis, but to encourage you to find time to connect with us on #toolchain, so we might give a more conversational breakdown of your problems and find workable solutions that work for you, that work for CPAN, and make CPAN authors in general happy.

Because having your code standardised and vulcanised to CPAN standards will work in your favour for seeing adoption. It will work in your favor for seeing your module get tested and having the bugs fixed.

It will work in your favour to help find optimal places to fragment your code to be optimally consumable, which will make your life easier long term by making it more amenable to contribution, simplifying the logic you need to internalise.

( PS. Please don't call me "Fredrick" or "Fredric", the former is not any part of my name, and the latter is stricly a surname and I find being addressed by it as a title implicitly offensive. Its an easy enough mistake for people to make, but I can't not point it out ;) )

Sun Mar 08 11:10:16 2015 KENTNL [...] cpan.org - Cc KENTNL added

Wed Mar 11 08:06:51 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

Subject:	Re: [rt.cpan.org #100658] documentation lies
Date:	Wed, 11 Mar 2015 08:06:18 -0400
To:	bug-Net-FullAuto [...] rt.cpan.org
From:	brian.kelly [...] fullautosoftware.net

Thanks for the reply Kent. And - yes, apologies for my "mistake". Your advice is unassailable. I intend to take advantage of it. Thanks for you thoughtful reply. Brian Kelly Show quoted text

> Queue: Net-FullAuto > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=100658 > > > My intention with showing the sample solution I did was to expose you to > the > sorts of solutions and knowledge that can be gained from the CPAN > community, > specifically toolchain operators, who have combined experience of well > over 100 > years =). > > The suggestion was not so much to take the suggestion gratis, but to > encourage > you to find time to connect with us on #toolchain, so we might give a more > conversational breakdown of your problems and find workable solutions that > work > for you, that work for CPAN, and make CPAN authors in general happy. > > Because having your code standardised and vulcanised to CPAN standards > will > work in your favour for seeing adoption. It will work in your favor for > seeing > your module get tested and having the bugs fixed. > > It will work in your favour to help find optimal places to fragment your > code > to be optimally consumable, which will make your life easier long term by > making it more amenable to contribution, simplifying the logic you need to > internalise. > > ( PS. Please don't call me "Fredrick" or "Fredric", the former is not any > part > of my name, and the latter is stricly a surname and I find being addressed > by > it as a title implicitly offensive. Its an easy enough mistake for people > to > make, but I can't not point it out ;) ) > >

Mon Mar 16 12:03:19 2015 tlinden [...] cpan.org - Correspondence added

PS: here's an example of how to do it right: https://www.rexify.org/.

Tue Mar 17 08:56:15 2015 Brian.Kelly [...] fullautosoftware.net - Correspondence added

Subject:	Re: [rt.cpan.org #100658] documentation lies
Date:	Tue, 17 Mar 2015 08:55:16 -0400
To:	bug-Net-FullAuto [...] rt.cpan.org
From:	brian.kelly [...] fullautosoftware.net

Thanks, I'll check it out. Show quoted text

> Queue: Net-FullAuto > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=100658 > > > PS: here's an example of how to do it right: https://www.rexify.org/. >

Sun Mar 13 17:52:30 2016 Brian.Kelly [...] fullautosoftware.net - Correspondence added

RT-Send-CC:

ether [...] cpan.org, ej_zg [...] hotmail.com

On Tue Dec 02 05:11:20 2014, TLINDEN wrote: Show quoted text

I will address each item one at a time: Show quoted text

>> - requests root permissions here and there

Most of this type of code has been moved out and put in different modules that are specifically written for setting up NEW servers launched by FullAuto via the AWS EC2 API. When setting up a new server with software, needing root level privileges is unavoidable. Users need to provide credentials that have Administrator privileges activated for any of this to work on AWS EC2. There is no way that any of this code can be successfully used without the full knowledge and buy-in of those using it. Show quoted text

>> - installs and removes software (local and remote) like apache, mysql

and so forth That code is in a separate module called an "Instruction Set". It's name is Liferay_is.pm and it is a module that "instructs" FullAuto in launching 3 servers via the EC2 AWS API and then provisioning those servers in turn, and finally cross configuring them, and starting up the entire architecture. This is an automated standup of a complete environment - NOT FROM PRE-BUILT IMAGES, but from source code and repositories. That's what FullAuto does, and it does it via persistent SSH and SFTP connections to these servers. Right now there are 5 "demo" "instructions sets" bundled with the distribution. Later these will be separated from the core bundle, and released independently. There is also an instruction set that sets up a four node Hadoop cluster. There is one that sets up a full working Catalyst server that will soon serve as a platform for coupling a RESTful API to FullAuto. I fail to see how any of this can viewed as a "problem". FullAuto was built to automate workloads like Chef, Puppet, Salt, Ansible and others do, with a LOT less overhead and more flexibility and ease of use. Show quoted text

>> - maintains amazon ec2 and aws instances

NO - it does not. The demo "instruction sets" do launch from generic images, bu t there is no image "maintenance". All image manipulation is done from code in the instruction set - NOT the core engine. Show quoted text

>> - creates and deletes files and directories in remote and local directories

Again - this is code found in the "instruction sets". Manipulating files is a common workload automation task. Again, I fail to see how this can be an issue? Show quoted text

>> - calls tar, transfers tarballs, unpacks and removes them on various

occurences in the code Again - this code is now mostly found in instruction sets. However, there is code in the core that does dynamic mirroring of directories using a combination of sftp and ssh. It does a diff via "ls" output and bundles just the files that have changed, and does use tar to move those in order speed up the transfer. This functionality is currently undocumented, but will be documented soon. Again I fail to see how this is a "problem". Show quoted text

>> creates/modifies crontab entries

Again, an undocumented and not completely implemented feature. Not likely to be elevated anytime soon. Show quoted text

>> - lot of more stuff I didn't have the time even trying to understand

What "standard" are you suggesting? That it can be completely digested in 10 minutes, 10 hours or 10 days? The code is ALL there. Nothing is hidden. There is a LOT of it, but most of the superfluous stuff has been removed. What remains actually works and is a core feature - or is close to being one. Show quoted text

>> While the documentation talks about security, the module nevertheless

> stores passwords to berkeley db files. Yes, those are encrypted, but > what about the encryption key? It's stored elsewhere, since this > thing's main purpose is to run from cron. Obviously the author hasn't > ever heard of ssh keys.

Nonsense - of course I've heard of ssh keys, and the updated POD documentation strongly advises their use. Nevertheless, passwords are still prevalent, and if that is all that is available, then they can be used. And yes, the berkeley db treatment is hardly a "perfect" solution, but it beats storing passwords in cleartext. It's a compromise - and not even a very good one, but again, better than cleartext. I strongly urge the use of keys and identity files - and do frequently in the documentation. Using keys and identity files is supported and easy to use with FullAuto, and is documented, whereas the berkeley db stuff currently isn't. Show quoted text

>> There are 174 subs, many of them are duplicates

> (e.g. there are 4 subs "cwd").

Show quoted text

>> Those "cwd"s are in different packages, and serve different purposes, but all designed to give the user a seamless and singular interface experience. The one in the Remote_CMD package deals specifically with SSH and TELNET sessions. The one in File_Transfer deals with SFTP and FTP sessions. The one in MAIN deals with localhost and one in "Broken" allows for control over error conditions. None of it is "cruft".

Show quoted text

>> There are 36 packages within FA_Core.pm, most of them declared inside subs.

Those are subs designed to work with Term::Menus and declaring a unique package within them prevents namespace collisions which was a big problem for me early on. These routines are subject to a form of code injection using Data::Dump::Streamer - which can see inside of closures. Nothing else at the time could that I was aware of. These subroutines need to be "persisted" in memory for dynamic menu creation. Show quoted text

>> About 20-30% of the

> code is redundant, i.e. duplicated all over the module.

I am guilty of some of this - and have been working to reduce a lot of this as time and opportunity permits. But it's not nearly as bad as you suggest. Show quoted text

>> The file is soo large, that it is impossible to understand for an outsider.

Not "impossible" - that's absurd. Difficult? Yes - I admit that. And I can appreciate that you lack the incentive to do more than a cursory examination. However, I have NO doubt, that if I've built something compelling that someone, and perhaps many, WILL take the time to tear it down and learn what it does and how it does it. The proof as they say - is in the pudding. Show quoted text

>> The module should - if any - be rewritten from scratch. In its current

state it can't be trusted. Nonsense. Could it benefit from "refactoring" - yes of course, and that is an ongoing activity. The thing works - it works well, it performs well, it reports errors extremely well. I would not call that something that needs to be "rewritten from scratch". And as far as it being "trustworthy" - that is a call everyone is entitled to make on their own. FullAuto is AMBITIOUS and highly unique software, and no amount of refactoring is going magically turn it into something an outsider can digest in 10 minutes. The whole "persistent connection" capability is something that I am unaware of anyone else successfully using. Chef, Puppet, Salt all use remote agents. Ansible "claims" to be "agent-less" but in fact has a dependency on Python being on remote nodes. And it also has a dependency on OpenSSH persistConnect feature being activated. FullAuto has NONE of these dependencies, but can perform every bit as powerfully and perform the same kind of workloads no matter the size or complexity - and can do it with micro images in any cloud environment, and NO dependency requirements for the remote nodes. If you know of another framework that can boast that same capability - I'm all ears! One final thing - this project has been ongoing since 2000. My Perl skills are a 1000 times greater than what they were when I started the project. Would I do things differently if I was starting over? Certainly. But I'm not starting over, and if a code construct works, works well, and solves a problem even if it's not a "best practice" way to do it, I'm going to stick with it for the "moment" and refactor it only when it becomes convenient to do so. I have a lot of other things to attend to that have higher priority - unfortunately.

Sun Mar 13 17:52:32 2016 Brian.Kelly [...] fullautosoftware.net - Status changed from 'open' to 'resolved'

Sun Mar 13 17:52:33 2016 Brian.Kelly [...] fullautosoftware.net - Fixed in 1.0000102 added

Wed Sep 21 16:12:50 2016 tlinden [...] cpan.org - Status changed from 'resolved' to 'open'

Wed Sep 21 16:28:54 2016 tlinden [...] cpan.org - Correspondence added

Subject:

documentation lies - still

Still. Nothing has changed. Still multiple "releases" each day. Still a mess. Still unsecure/dangerous. Still lots of hardcoded stuff. Still not a framework.Makefile.PL still not following guidelines. Still using sudo. Still without asking. Still doing chmods on users system files all over the disk. Please.