Bug #103205 for Net-IDN-Encode: conversion of domain name xn--zcaa.de

Mon Mar 30 10:20:07 2015 ab [...] lixutec.net - Ticket created

Subject:

conversion of domain name xn--zcaa.de

the conversion of the IDN domain name "xn--zcaa.de" to unicode is broken: The expected result would be "ßß.de". But using the domain_to_unicode() method a wrongly encoded string is returned. The issue only appears if the domain name only contains "ß" characters. If the domain name also contains other characters the returned unicode string is correct.

Mon Mar 30 15:37:43 2015 CFAERBER [...] cpan.org - Correspondence added

Thank you for reporting this bug. I cannot reproduce it on perl 5.16 with the latest version of Net-IDN-Encode. Which version of perl and Net-IDN-Encode are you using. However, I also have an idea what might be the cause. Could you please test the developer release at https://metacpan.org/release/CFAERBER/Net-IDN-Encode-2.201_20150330 and check whether the problem persists?

Mon Mar 30 15:37:43 2015 The RT System itself - Status changed from 'new' to 'open'

Tue Mar 31 04:50:50 2015 ab [...] lixutec.net - Correspondence added

From:

ab [...] lixutec.net

Show quoted text

> I cannot reproduce it on perl 5.16 with the latest version of Net-IDN- > Encode. Which version of perl and Net-IDN-Encode are you using.

I'm using two perl versions on different servers: v5.10.1 and v5.14.2 the version of Net-IDN-Encode is 2.201 I will try the development version soon...

Tue Mar 31 06:12:39 2015 ab [...] lixutec.net - Correspondence added

From:

ab [...] lixutec.net

Show quoted text

> However, I also have an idea what might be the cause. Could you please > test the developer release at > https://metacpan.org/release/CFAERBER/Net-IDN-Encode-2.201_20150330 > and check whether the problem persists?

The problem still exists with the latest developer release.

Tue Mar 31 07:24:38 2015 ab [...] lixutec.net - Correspondence added

From:

ab [...] lixutec.net

I wrote a short test script to isolate the issue: #!/usr/bin/perl use Net::IDN::Encode; print STDERR "VERSION: " . $Net::IDN::Encode::VERSION . "\n"; use Net::IDN::Encode ':all'; my $u = domain_to_unicode('xn--zcaa.de'); utf8::encode($u) if utf8::is_utf8($u); print STDERR "UNICODE: " . $u . "\n"; $u = domain_to_unicode('xn--m-qfaaa.de'); utf8::encode($u) if utf8::is_utf8($u); print STDERR "UNICODE: " . $u . "\n"; the output is: VERSION: 2.2012015033 UNICODE: ��.de UNICODE: mßßß.de

Tue Mar 31 08:37:31 2015 ab [...] lixutec.net - Correspondence added

From:

ab [...] lixutec.net

Show quoted text

> the output is: > VERSION: 2.2012015033 > UNICODE: ��.de > UNICODE: mßßß.de

BTW: using an older version works: VERSION: 2.003 UNICODE: ßß.de UNICODE: mßßß.de

Tue Mar 31 19:41:30 2015 CFAERBER [...] cpan.org - Correspondence added

Your problem is this line: utf8::encode($u) if utf8::is_utf8($u); This is wrong. If your display requires UTF-8, you need to encode both strings encoded in UTF-X (is_utf8 is true) as well as strings encoded as bytes (is_utf8 is false). For output, you can also use e.g binmode STDERR, ':utf8'; I'm don't think that Net::IDN::Encode should guarantee the status of the utf8 flag on *_to_unicode. Normally, this flag is handled seamlessly by perl.

Tue Mar 31 19:41:31 2015 CFAERBER [...] cpan.org - Status changed from 'open' to 'rejected'

Tue Mar 31 19:41:32 2015 CFAERBER [...] cpan.org - Taken

Thu Apr 02 15:02:19 2015 CFAERBER [...] cpan.org - Correspondence added

Please also note the following from perlunifaq: ------------------------------------------------------------------------------- What is "the UTF8 flag"? Please, unless you're hacking the internals, or debugging weirdness, don't think about the UTF8 flag at all. That means that you very probably shouldn't use is_utf8 , _utf8_on or _utf8_off at all.