Skip Menu |

This queue is for tickets about the Net-SSLeay CPAN distribution.

Report information
The Basics
Id: 98368
Status: resolved
Priority: 0/
Queue: Net-SSLeay

People
Owner: MIKEM [...] cpan.org
Requestors: victor [...] vsespb.ru
Cc: ether [...] cpan.org
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Input data should be utf8::downgrad'ed
==== use strict; use warnings; use Net::SSLeay; use Data::Dumper; use Digest::MD5 qw/md5_hex/; use Digest::SHA qw/sha256_hex/; use MIME::Base64; my $msg= "\x80Hello"; utf8::upgrade($msg) if $ARGV[0]; print md5_hex($msg), "\n"; utf8::upgrade($msg) if $ARGV[0]; print sha256_hex($msg), "\n"; utf8::upgrade($msg) if $ARGV[0]; print encode_base64($msg), "\n"; utf8::upgrade($msg) if $ARGV[0]; my ($page, $response, %reply_headers) = Net::SSLeay::post_https( 'posttestserver.com', # see http://www.posttestserver.com/ 443, '/post.php', Net::SSLeay::make_headers( 'Content-Type' => 'text/xml; charset:utf-8', ), $msg ); print Dumper $page; === === perl test.pl dea59e28356a94aeca613cc57dee3f68 d26e7eab001349185c929fc5a009e2ae7a5177aef4984ab83d357d419e305360 gEhlbGxv $VAR1 = 'Successfully dumped 0 post variables. View it at http://www.posttestserver.com/data/2014/08/27/14.32.221601594009 Post body was 6 chars long.'; === === perl test.pl 1 dea59e28356a94aeca613cc57dee3f68 d26e7eab001349185c929fc5a009e2ae7a5177aef4984ab83d357d419e305360 gEhlbGxv $VAR1 = 'Successfully dumped 0 post variables. View it at http://www.posttestserver.com/data/2014/08/27/14.32.4362975749 Post body was 7 chars long.'; === In this example binary data which was utf8::upgrad'ed (UTF-8 flag on) behaves differently from utf8::downgraded one (see different length of body) there was a similar bug report here https://rt.cpan.org/Ticket/Display.html?id=81668 you closed the bug and documented the issue. but my point is that according to perl documentation downgraded and upgraded data should behave same way, and this issue is something that should be fixed. you can see in both example runs above that md5, sha, and base64 of the scalar are always same, no matter if it's upgraded or not. all perl functions behave similar way with utf-8 flagged strings. upgraed and downgraded forms of strings are same in perl (they match via "eq"). good article about UTF-8 flag: http://blogs.perl.org/users/aristotle/2011/08/utf8-flag.html So I propose utf8::downgrade any binary data coming to module (in post_https, ssl_write_all, write_partial and other functions)
Subject: Re: [rt.cpan.org #98368] Input data should be utf8::downgrad'ed
Date: Thu, 28 Aug 2014 07:59:08 +1000
To: bug-Net-SSLeay [...] rt.cpan.org
From: Mike McCauley <mikem [...] airspayce.com>
Hello, As you note, the documentation for post_https() says you should not pass UTF-8 data as content: use utf8::downgrade first. Unfortunately its not possible to automatically upgrade and downgrade strings inside net-ssleay, since utf::downgrade() etc is not natively available in all versions of perl that net-ssleay supports. I am *very* reluctant to add that dependency. So I do not expect the relevant code in net-ssleay and the warning about UTF strings in the documentation to change. You will have to downgrade in your calling code. Cheers. On Wednesday, August 27, 2014 05:42:26 PM you wrote: Show quoted text
> Wed Aug 27 17:42:24 2014: Request 98368 was acted upon. > Transaction: Ticket created by vsespb > Queue: Net-SSLeay > Subject: Input data should be utf8::downgrad'ed > Broken in: (no value) > Severity: (no value) > Owner: Nobody > Requestors: victor@vsespb.ru > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=98368 > > > > ==== > use strict; > use warnings; > use Net::SSLeay; > use Data::Dumper; > use Digest::MD5 qw/md5_hex/; > use Digest::SHA qw/sha256_hex/; > use MIME::Base64; > > my $msg= "\x80Hello"; > > > > utf8::upgrade($msg) if $ARGV[0]; > print md5_hex($msg), "\n"; > utf8::upgrade($msg) if $ARGV[0]; > print sha256_hex($msg), "\n"; > utf8::upgrade($msg) if $ARGV[0]; > print encode_base64($msg), "\n"; > utf8::upgrade($msg) if $ARGV[0]; > > my ($page, $response, %reply_headers) > = Net::SSLeay::post_https( > 'posttestserver.com', # see http://www.posttestserver.com/ > 443, > '/post.php', > Net::SSLeay::make_headers( > 'Content-Type' => 'text/xml; charset:utf-8', > ), > $msg > ); > print Dumper $page; > > === > > === > perl test.pl > dea59e28356a94aeca613cc57dee3f68 > d26e7eab001349185c929fc5a009e2ae7a5177aef4984ab83d357d419e305360 > gEhlbGxv > > $VAR1 = 'Successfully dumped 0 post variables. > View it at http://www.posttestserver.com/data/2014/08/27/14.32.221601594009 > Post body was 6 chars long.'; > === > > === > perl test.pl 1 > dea59e28356a94aeca613cc57dee3f68 > d26e7eab001349185c929fc5a009e2ae7a5177aef4984ab83d357d419e305360 > gEhlbGxv > > $VAR1 = 'Successfully dumped 0 post variables. > View it at http://www.posttestserver.com/data/2014/08/27/14.32.4362975749 > Post body was 7 chars long.'; > === > > In this example binary data which was utf8::upgrad'ed (UTF-8 flag on) > behaves differently from utf8::downgraded one (see different length of > body) > > there was a similar bug report here > https://rt.cpan.org/Ticket/Display.html?id=81668 you closed the bug and > documented the issue. but my point is that according to perl documentation > downgraded and upgraded data should behave same way, and this issue is > something that should be fixed. > > you can see in both example runs above that md5, sha, and base64 of the > scalar are always same, no matter if it's upgraded or not. all perl > functions behave similar way with utf-8 flagged strings. > > upgraed and downgraded forms of strings are same in perl (they match via > "eq"). > > good article about UTF-8 flag: > http://blogs.perl.org/users/aristotle/2011/08/utf8-flag.html > > So I propose utf8::downgrade any binary data coming to module (in > post_https, ssl_write_all, write_partial and other functions)
-- Mike McCauley VK4AMM mikem@airspayce.com Airspayce Pty Ltd 9 Bulbul Place Currumbin Waters QLD 4223 Australia http://www.airspayce.com Phone +61 7 5598-7474
ok, thx. then I'll try to report it to other modules that use your module, so they can workaround. On Thu Aug 28 01:59:21 2014, mikem@airspayce.com wrote: Show quoted text
> Hello, > > As you note, the documentation for post_https() says you should not > pass UTF-8 > data as content: use utf8::downgrade first. > > Unfortunately its not possible to automatically upgrade and downgrade > strings > inside net-ssleay, since utf::downgrade() etc is not natively > available in all > versions of perl that net-ssleay supports. I am *very* reluctant to > add that > dependency. > > So I do not expect the relevant code in net-ssleay and the warning > about UTF > strings in the documentation to change. You will have to downgrade in > your > calling code. > > Cheers. > > On Wednesday, August 27, 2014 05:42:26 PM you wrote:
> > Wed Aug 27 17:42:24 2014: Request 98368 was acted upon. > > Transaction: Ticket created by vsespb > > Queue: Net-SSLeay > > Subject: Input data should be utf8::downgrad'ed > > Broken in: (no value) > > Severity: (no value) > > Owner: Nobody > > Requestors: victor@vsespb.ru > > Status: new > > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=98368 > > > > > > > ==== > > use strict; > > use warnings; > > use Net::SSLeay; > > use Data::Dumper; > > use Digest::MD5 qw/md5_hex/; > > use Digest::SHA qw/sha256_hex/; > > use MIME::Base64; > > > > my $msg= "\x80Hello"; > > > > > > > > utf8::upgrade($msg) if $ARGV[0]; > > print md5_hex($msg), "\n"; > > utf8::upgrade($msg) if $ARGV[0]; > > print sha256_hex($msg), "\n"; > > utf8::upgrade($msg) if $ARGV[0]; > > print encode_base64($msg), "\n"; > > utf8::upgrade($msg) if $ARGV[0]; > > > > my ($page, $response, %reply_headers) > > = Net::SSLeay::post_https( > > 'posttestserver.com', # see http://www.posttestserver.com/ > > 443, > > '/post.php', > > Net::SSLeay::make_headers( > > 'Content-Type' => 'text/xml; charset:utf-8', > > ), > > $msg > > ); > > print Dumper $page; > > > > === > > > > === > > perl test.pl > > dea59e28356a94aeca613cc57dee3f68 > > d26e7eab001349185c929fc5a009e2ae7a5177aef4984ab83d357d419e305360 > > gEhlbGxv > > > > $VAR1 = 'Successfully dumped 0 post variables. > > View it at > > http://www.posttestserver.com/data/2014/08/27/14.32.221601594009 > > Post body was 6 chars long.'; > > === > > > > === > > perl test.pl 1 > > dea59e28356a94aeca613cc57dee3f68 > > d26e7eab001349185c929fc5a009e2ae7a5177aef4984ab83d357d419e305360 > > gEhlbGxv > > > > $VAR1 = 'Successfully dumped 0 post variables. > > View it at > > http://www.posttestserver.com/data/2014/08/27/14.32.4362975749 > > Post body was 7 chars long.'; > > === > > > > In this example binary data which was utf8::upgrad'ed (UTF-8 flag on) > > behaves differently from utf8::downgraded one (see different length > > of > > body) > > > > there was a similar bug report here > > https://rt.cpan.org/Ticket/Display.html?id=81668 you closed the bug > > and > > documented the issue. but my point is that according to perl > > documentation > > downgraded and upgraded data should behave same way, and this issue > > is > > something that should be fixed. > > > > you can see in both example runs above that md5, sha, and base64 of > > the > > scalar are always same, no matter if it's upgraded or not. all perl > > functions behave similar way with utf-8 flagged strings. > > > > upgraed and downgraded forms of strings are same in perl (they match > > via > > "eq"). > > > > good article about UTF-8 flag: > > http://blogs.perl.org/users/aristotle/2011/08/utf8-flag.html > > > > So I propose utf8::downgrade any binary data coming to module (in > > post_https, ssl_write_all, write_partial and other functions)
On 2014-08-27 14:59:21, mikem@airspayce.com wrote: Show quoted text
> Unfortunately its not possible to automatically upgrade and downgrade > strings > inside net-ssleay, since utf::downgrade() etc is not natively > available in all > versions of perl that net-ssleay supports. I am *very* reluctant to > add that > dependency.
What's wrong with only calling utf8::downgrade on versions that support it? i.e.: utf8::downgrade($payload) if eval { utf8->can('downgrade') };