Ah, this is a thing where the string looks like utf8 to you but is flat bytes to Perl.
Add this line to your code:
print "It is ", length($str), " characters long.\n";
And it'll say
It is 33 characters long.
But use this program:
#!/usr/bin/perl
use Text::Unidecode;
# copied from a website where it appears as value="\"@timestamp\":\""
my $str = 'value=”\”@timestamp\”:\”"';
utf8::decode($str);
binmode(STDOUT, ":utf8") || die "WHUT $!";
# Read perldoc:
# perlunitut, perluniintro, perlrun, bytes, perlunicode perluni
# where there's explanations of perl -CL and other fun stuff
# that might, or might not, be more DWIM than having to
# call utf8::decode as above.
print 'string as it appears on website : value="\"@timestamp\":\""' . "\n";
print "raw string as copy/pasted in Mac terminal: $str\n";
print "It is ", length($str), " characters long.\n";
print "string returned by unidecode() : " . unidecode($str) . "\n";
And that works, and it says:
It is 25 characters long.
string returned by unidecode() : value="\"@timestamp\":\""
The "a"s were coming from the fact that the byte values for the ” you have is e2 80 9d.
Now, 80 and 9d are no good in Unicode so each of them are empty-string, but e2 is "â"
...which Unidecode turns into "a", and that's why it looks like Unidecode is
turning a “ character into an a character.
BTW, in mystery cases like this, I often throw in a thing like this to make
sure that what I consider characters and what Perl considers characters are
syncing up, or not:
foreach my $char (split '', $str) {
printf "\tChar %0x : \"%s\" => u:\"%s\"\n",
ord($char), $char, unidecode($char);
}
Am I making sense? I often explain things poorly and can't tell.
And "perldoc utf8" sometimes leaves me more confused than before I read it!
I often just go thru the various functions and call one or the other until
I get whichever one does the job... and then I see that its documentation
*now* (in 20/20 hindsight) makes perfect sense.
OH UNICODE!
