You have in part brought your problem on yourself. The line:
Show quoted text> carp $encoded_utf8;
is requesting output of the characters corresponding to the octets of
the UTF-8 encoding. These are characters from the high half of Latin-1,
not the characters that the UTF-8 octets represent. You have asked for
mojibake, and shouldn't be surprised that you got it.
However, another part of your situation comes from awkward behaviour
of Perl. By default, Perl treats a textual I/O handle as using the
Latin-1 encoding. This means that a string containing non-Latin-1
characters cannot be correctly emitted. But if an attempt is made to
output such a string, Perl does not treat it as a fatal error; instead, it
treats the output handle as using UTF-8 encoding *just for that string*.
It also emits the warning "Wide character" to tell you it's doing that.
I think, from your description of the output, that your terminal is
actually expecting UTF-8. This means that Perl's default treatment of
output handles (Latin-1, remember) is not correct for you. Wherever you
use non-ASCII characters, you *should* tell Perl what encoding to use on
I/O handles. That you do not is a bug: you are not using Perl correctly.
Now, I'll examine the output you get. In the feedme_encoded_utf8 case,
the sub's parameter is a string of 9 octets with their high bit set, and
$decoded_utf8 ends up containing three katakana characters. When you
try to output that string as a warning, these characters can't go into
the output stream properly, because they have no encoding in Latin-1.
So Perl issues a warning and then outputs their UTF-8 encoding instead.
The UTF-8 happens to be what your terminal is expecting, so your terminal
shows you the katakana. The two bugs (yours and Perl's) cancel out, and
so by accident you see the string Perl was actually trying to show you,
which is the katakana.
In the feedme_decoded_utf8 case, the sub's parameter is a string of three
katakana characters, and $encoded_utf8 ends up containing 9 octets with
their high bit set, or equivalently 9 characters from the upper half of
Latin-1 (mostly accented Latin letters). When you try to output that as
a warning, this 9-character string that you supplied gets concatenated
with the stack trace. For murky historical reasons, the sub's parameter
doesn't get quoted in this case, so the stack trace contains the actual
katakana characters. So the complete warning string includes those
accented Latin letters, and (further on) three katakana characters.
As before, the katakana won't fit into Latin-1 encoding for output, so,
as before, Perl outputs the warning message encoded in UTF-8 instead.
This means that, as before, by accident you correctly see the string Perl
was actually trying to show you. You see the mojibake that you asked for.
If Carp is amended to quote all subroutine arguments, which is essentially
what you suggest, this changes the feedme_decoded_utf8 case. With the
sub's parameter represented in ASCII, the complete warning string
now does not contain any katakana. Of course, it still contains the
9 high-half Latin-1 characters, the mojibake, that you supplied as
the primary message. Now the entire warning string can be encoded
in Latin-1, so Perl does encode it that way for output, and does not
emit a "Wide character" warning. But because your terminal's character
encoding is mismatched with Perl's treatment of the output handle, you
don't correctly see the mojibake that you asked for. Instead, because
your terminal is expecting UTF-8, it accidentally undoes the mojibake,
and shows you the three katakana characters. Although you preferred
this output, it is not correct.
I think it *is* sensible for Carp to escape all subroutine arguments,
representing them entirely in ASCII. But this is not for your reasons.
-zefram
