Skip Menu |

This queue is for tickets about the Net-Amazon-EC2 CPAN distribution.

Report information
The Basics
Id: 127409
Status: open
Priority: 0/
Queue: Net-Amazon-EC2

People
Owner: Nobody in particular
Requestors: jamesa [...] peakpayment.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Bug in Net::Amazon::EC2
Date: Fri, 19 Oct 2018 15:26:44 -0600
To: bug-Net-Amazon-EC2 [...] rt.cpan.org, Scott Welling <scottw [...] peakpayment.com>
From: James Alton <jamesa [...] peakpayment.com>
Dear Net::Amazon::EC2 module maintainer: I think we found a weird bug where AWS Auth errors are produced. I've never seen these in the past, so I reverted versions until I found it was working without Auth problems in 0.34 and failed in 0.35/0.36. (I only got AWS Auth problems about half the time I ran my code, randomly.) - Distribution name and version: Net::Amazon::EC2 0.35 and 0.36 (bug is not in 0.34) - Perl version (Find this by running the command perl -v): This is perl 5, version 24, subversion 1 (v5.24.1) built for x86_64-linux-gnu-thread-multi (with 81 registered patches, see perl -V for more detail) - Operating System vendor and version (Find this by running the command uname -a): Linux server 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u5 (2018-09-30) x86_64 GNU/Linux - Exact cut and pasted error or warning messages: Amazon EC2 Errors [Request 9156b930-6025-4803-ba92-85c22f705d44]: [AuthFailure] AWS was not able to validate the provided access credentials - The shortest, clearest code you can manage to write which reproduces the bug described. use Net::Amazon::EC2; my $ec2 = Net::Amazon::EC2->new( AWSAccessKeyId => '...', SecretAccessKey => '...', signature_version => 4, ); my $descr_snap = $ec2->describe_snapshots(Owner => 'self'); # Auth error happens on executing the above line. Half the time there is an Auth error, half the time it works. Reverting to Net::Amazon::EC2 version 0.34 makes the AWS Auth error go away. Theories: The ->describe_snapshots() call above takes a few seconds. Maybe the HMAC stuff has expired somehow in an unexpected way making failure happen? (Not sure why the difference in 0.34 vs 0.35/0.36.) Here is a dump of an Auth failure when debug => 1 was added: (with lots of info redacted) Attempting to fetch instance credentials CONTENT: Action=DescribeSnapshots&Owner=self&Version=2014-06-15&X-Amz-Security-Token=------------------------ CANONICAL REQUEST: POST / content-type:application/x-www-form-urlencoded host:ec2.us-east-1.amazonaws.com x-amz-date:20181019T203749Z x-amz-security-token:------------------------------------- content-type;host;x-amz-date;x-amz-security-token ------------------------------------------- STRING TO SIGN: AWS4-HMAC-SHA256 20181019T203749Z 20181019/us-east-1/ec2/aws4_request ---------------------------------- HTTP REQUEST: POST https://ec2.us-east-1.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=-------------------/20181019/us-east-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=---------------------------------------- Content-Type: application/x-www-form-urlencoded X-Amz-Date: 20181019T203749Z X-Amz-Security-Token: ---------------------------------- Action=DescribeSnapshots&Owner=self&Version=2014-06-15&X-Amz-Security-Token=----------------------------------------- $VAR1 = { 'RequestID' => '40372ec9-9100-4583-ab14-04f59fdf18ef', 'Errors' => [ { 'Error' => { 'Message' => 'AWS was not able to validate the provided access credentials', 'Code' => 'AuthFailure' } } ] }; ERROR CODE: AuthFailure MESSAGE: AWS was not able to validate the provided access credentials FOR REQUEST: 40372ec9-9100-4583-ab14-04f59fdf18ef Last error was: AWS was not able to validate the provided access credentials at /usr/local/share/perl/5.24.1/Net/Amazon/EC2.pm line 484. Net::Amazon::EC2::_parse_errors(Net::Amazon::EC2=HASH(0x563bbb9fcbd8), HASH(0x563bbf5bfee8)) called at /usr/local/share/perl/5.24.1/Net/Amazon/EC2.pm line 3046 Net::Amazon::EC2::describe_snapshots(undef, "Owner", "self") called at /usr/local/bin/ec2_snapshot_cleanup.pl line 40 (Line 40 of our ec2_snapshot_cleanup.pl code had the "->describe_snapshots()" call) Thank you, James Alton
Hi James I'm using a Ubuntu 18.04 LTS and have tried to reproduce your findings regarding the Net::Amazon::EC2 module. To do this, I've cloned the upstream repository git clone https://github.com/mrallen1/net-amazon-ec2 I am talking about commit ceeb817cc which says it was just bumped to version 0.36 of the module. Then I've tried running your code snippet as a oneliner as follows perl -I./lib -MData::Dumper -MNet::Amazon::EC2 -e 'my $ec2 = Net::Amazon::EC2->new(AWSAccessKeyId=>"...",SecretAccessKey => "...",signature_version => 4); my $descr_snap = $ec2->describe_snapshots(Owner=>"self"); print Dumper $descr_snap; ' It ran without problems, but gave me no results. I then removed Owner=>"self" and re-ran it, I got a big object back with a lot of different snapshots. I tried reproducing the problem by following your assumption where you say this might have happened because the network was slow and the HMAC might have expired. So, according to [1] [2], I've enabled the ifb module, and I've also installed wondershaper, which is just a wrapper that runs "tc qdisc" commands to help simulate bandwidth limitations. I ran the following as root apt-get install wondershaper modprobe ifb wondershaper eth0 300 300 tc qdisc add dev eth0 root netem loss 25% 25% (the two numbers for packet loss are better explained in [2]) My understanding is that this limits my downlink speed to 300kb and uplink speed to 300kb and then I've also inserted some packet loss using the tc command. I then reran the oneliner, it still worked, and I didn't see any error like the one you mentioned in your bug report. I then cleared these bandwidth limitations from my network interface wondershaper clear eth0 I also did a quick review of the differences between 0.34 and 0.35 like this git diff 0.34 0.35 Nothing caught my eye except some header names with different casing such as "X-Amz-Security-Token" in 0.34 which was replaced with "x-amz-security-token" in 0.35. So I was not able to reproduce your findings. Maybe more details would help [1] https://www.excentis.com/blog/use-linux-traffic-control-impairment-node-test-environment-part-2 [2] https://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux On Fri Oct 19 17:27:13 2018, jamesa@peakpayment.com wrote: Show quoted text
> Dear Net::Amazon::EC2 module maintainer: > > I think we found a weird bug where AWS Auth errors are produced. I've > never > seen these in the past, so I reverted versions until I found it was > working > without Auth problems in 0.34 and failed in 0.35/0.36. (I only got AWS > Auth > problems about half the time I ran my code, randomly.) > > - Distribution name and version: Net::Amazon::EC2 0.35 and 0.36 (bug > is > not in 0.34) > - Perl version (Find this by running the command perl -v): This is > perl > 5, version 24, subversion 1 (v5.24.1) built for > x86_64-linux-gnu-thread-multi (with 81 registered patches, see perl -V > for > more detail) > - Operating System vendor and version (Find this by running the > command uname > -a): Linux server 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u5 > (2018-09-30) x86_64 GNU/Linux > > > - Exact cut and pasted error or warning messages: > Amazon EC2 Errors [Request 9156b930-6025-4803-ba92-85c22f705d44]: > [AuthFailure] AWS was not able to validate the provided access > credentials > - The shortest, clearest code you can manage to write which reproduces > the bug described. > use Net::Amazon::EC2; > my $ec2 = Net::Amazon::EC2->new( > AWSAccessKeyId => '...', > SecretAccessKey => '...', > signature_version => 4, > ); > my $descr_snap = $ec2->describe_snapshots(Owner => 'self'); > # Auth error happens on executing the above line. Half the time there > is > an Auth error, half the time it works. Reverting to > Net::Amazon::EC2 version 0.34 makes the AWS Auth error go away. > > Theories: The ->describe_snapshots() call above takes a few seconds. > Maybe the HMAC stuff has expired somehow in an unexpected way making > failure happen? (Not sure why the difference in 0.34 vs 0.35/0.36.) > > Here is a dump of an Auth failure when debug => 1 was added: (with > lots of > info redacted) > > Attempting to fetch instance credentials > CONTENT: > Action=DescribeSnapshots&Owner=self&Version=2014-06-15&X-Amz-Security- > Token=------------------------ > > > > CANONICAL REQUEST: POST > / > > content-type:application/x-www-form-urlencoded > host:ec2.us-east-1.amazonaws.com > x-amz-date:20181019T203749Z > x-amz-security-token:------------------------------------- > > content-type;host;x-amz-date;x-amz-security-token > ------------------------------------------- > > > > STRING TO SIGN: AWS4-HMAC-SHA256 > 20181019T203749Z > 20181019/us-east-1/ec2/aws4_request > ---------------------------------- > > > > HTTP REQUEST: POST https://ec2.us-east-1.amazonaws.com > Authorization: AWS4-HMAC-SHA256 > Credential=-------------------/20181019/us-east-1/ec2/aws4_request, > SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, > Signature=---------------------------------------- > Content-Type: application/x-www-form-urlencoded > X-Amz-Date: 20181019T203749Z > X-Amz-Security-Token: ---------------------------------- > > Action=DescribeSnapshots&Owner=self&Version=2014-06-15&X-Amz-Security- > Token=----------------------------------------- > > > > > > $VAR1 = { > 'RequestID' => '40372ec9-9100-4583-ab14-04f59fdf18ef', > 'Errors' => [ > { > 'Error' => { > 'Message' => 'AWS was not able to > validate the provided access credentials', > 'Code' => 'AuthFailure' > } > } > ] > }; > > > ERROR CODE: AuthFailure MESSAGE: AWS was not able to validate the > provided > access credentials FOR REQUEST: 40372ec9-9100-4583-ab14-04f59fdf18ef > > > > Last error was: AWS was not able to validate the provided access > credentials at /usr/local/share/perl/5.24.1/Net/Amazon/EC2.pm line > 484. > Net::Amazon::EC2::_parse_errors(Net::Amazon::EC2=HASH(0x563bbb9fcbd8), > HASH(0x563bbf5bfee8)) called at > /usr/local/share/perl/5.24.1/Net/Amazon/EC2.pm line 3046 > Net::Amazon::EC2::describe_snapshots(undef, "Owner", "self") called at > /usr/local/bin/ec2_snapshot_cleanup.pl line 40 > > (Line 40 of our ec2_snapshot_cleanup.pl code had the > "->describe_snapshots()" call) > > Thank you, > James Alton