Skip Menu |

Preferred bug tracker

Please visit the preferred bug tracker to report your issue.

This queue is for tickets about the YAML CPAN distribution.

Report information
The Basics
Id: 25434
Status: resolved
Priority: 0/
Queue: YAML

People
Owner: Nobody in particular
Requestors: dolmen [...] cpan.org
SREZIC [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in:
  • 0.62
  • 0.63
  • 0.64
  • 0.65
  • 0.66
  • 0.67
  • 0.68
  • 0.69_01
  • 0.69_02
  • 0.70
Fixed in: 0.71



Subject: Unicode/utf8 support missing
It seems that YAML cannot handle unicode data (codepoints > 255) when dumping to or loading from files. A roundtrip with data containing high unicode characters fails, as it could be seen in the attached script. Something like a $YAML::Syck::ImplicitUnicode variable would be nice. Regards, Slaven
Subject: yaml.pl
#!/usr/bin/perl use strict; use warnings; use Storable qw(); use YAML qw(); use YAML::Syck qw(); use Data::Dumper qw(); use Data::Compare qw(Compare); for my $bla (["äöü"], ["\x{20ac}"], ) { YAML::DumpFile("/tmp/yaml.yml", $bla); warn "YAML: " . Compare(YAML::LoadFile("/tmp/yaml.yml"), $bla) . "\n"; YAML::Syck::DumpFile("/tmp/yamlsyck.yml", $bla); warn "YAML::Syck: " . Compare(YAML::Syck::LoadFile("/tmp/yamlsyck.yml"), $bla) . "\n"; { local $YAML::Syck::ImplicitUnicode = 1; YAML::Syck::DumpFile("/tmp/yamlsyckutf8.yml", $bla); warn "YAML::Syck with ImplicitUnicode: " . Compare(YAML::Syck::LoadFile("/tmp/yamlsyckutf8.yml"), $bla) . "\n"; } Storable::nstore($bla, "/tmp/st.st"); warn "Storable: " . Compare(Storable::retrieve("/tmp/st.st"), $bla) . "\n"; }
Subject: [PATCH] Unicode/utf8 support missing
Here is a fix, including test cases. Note that fixing this bug fixes also RT43764, a Module::Build bug related to encoding problem in META.yml generation.
# Fix for bug 25434: utf8 support # # To apply this patch: # STEP 1: Chdir to the source directory. # STEP 2: Run the 'applypatch' program with this patch file as input. # # If you do not have 'applypatch', it is part of the 'makepatch' package # that you can fetch from the Comprehensive Perl Archive Network: # http://www.perl.com/CPAN/authors/Johan_Vromans/makepatch-x.y.tar.gz # In the above URL, 'x' should be 2 or higher. # # To apply this patch without the use of 'applypatch': # STEP 1: Chdir to the source directory. # If you have a decent Bourne-type shell: # STEP 2: Run the shell with this file as input. # If you don't have such a shell, you may need to manually create # the files as shown below. # STEP 3: Run the 'patch' program with this file as input. # # These are the commands needed to create/delete files/directories: # touch "t/dump-file-utf8.t" chmod 0666 "t/dump-file-utf8.t" # # This command terminates the shell and need not be executed manually. exit # #### End of Preamble #### #### Patch data follows #### diff -urp --binary "YAML-0.68\MANIFEST" "YAML-0.68.RT25434\MANIFEST" Index: ./MANIFEST --- ./MANIFEST Mon Dec 1 08:04:28 2008 +++ ./MANIFEST Thu May 7 10:15:00 2009 @@ -42,6 +42,7 @@ t/dump-basics.t t/dump-blessed.t t/dump-code.t t/dump-file.t +t/dump-file-utf8.t t/dump-nested.t t/dump-opts.t t/dump-perl-types.t diff -urp --binary "YAML-0.68\lib\YAML.pm" "YAML-0.68.RT25434\lib\YAML.pm" Index: ./lib/YAML.pm --- ./lib/YAML.pm Thu Dec 4 11:01:17 2008 +++ ./lib/YAML.pm Wed May 6 17:43:43 2009 @@ -52,9 +52,9 @@ sub DumpFile { if ($filename =~ /^\s*(>{1,2})\s*(.*)$/) { ($mode, $filename) = ($1, $2); } - open $OUT, $mode, $filename + open $OUT, "$mode:utf8", $filename or YAML::Base->die('YAML_DUMP_ERR_FILE_OUTPUT', $filename, $!); - } + } local $/ = "\n"; # reset special to "sane" print $OUT Dump(@_); } @@ -66,7 +66,7 @@ sub LoadFile { $IN = $filename; } else { - open $IN, $filename + open $IN, '<:utf8', $filename or YAML::Base->die('YAML_LOAD_ERR_FILE_INPUT', $filename, $!); } return Load(do { local $/; <$IN> }); diff -urp --binary nul "YAML-0.68.RT25434\t\dump-file-utf8.t" Index: ./t/dump-file-utf8.t --- ./t/dump-file-utf8.t Thu Jan 1 01:00:00 1970 +++ ./t/dump-file-utf8.t Thu May 7 10:12:06 2009 @@ -0,0 +1,36 @@ +use utf8; +use Test::YAML(); +BEGIN { + @Test::YAML::EXPORT = + grep { not /^(Dump|Load)(File)?$/ } @Test::YAML::EXPORT; +} +use t::TestYAML tests => 6; + +use YAML qw/DumpFile LoadFile/; + +ok defined &DumpFile, + 'DumpFile exported'; + +ok defined &LoadFile, + 'LoadFile exported'; + +my $file = 't/dump.yaml'; + +# A scalar containing non-ASCII characters +my $data = 'Olivier Mengué'; +is length($data), 14, 'Test source is correctly encoded'; + +DumpFile($file, $data); + +ok -e $file, + 'Output file exists'; + +open IN, '<:utf8', $file or die $!; +my $yaml = join '', <IN>; + +is $yaml, "--- $data\n", 'DumpFile YAML encoding is correct'; + +unlink $file; + +my $read = LoadFile($file); +is $read, $data, 'LoadFile is ok'; diff -urp --binary "YAML-0.68\t\meta-yml.t" "YAML-0.68.RT25434\t\meta-yml.t" Index: ./t/meta-yml.t --- ./t/meta-yml.t Thu Dec 4 11:06:34 2008 +++ ./t/meta-yml.t Thu May 7 09:59:32 2009 @@ -1,3 +1,4 @@ +use utf8; use t::TestYAML tests => 1; my $node_from_yaml = LoadFile('./META.yml'); #### End of Patch data #### #### ApplyPatch data follows #### # Data version : 1.0 # Date generated : Thu May 7 10:17:05 2009 # Generated by : makepatch 2.03 # Recurse directories : Yes # Excluded files : (\A|/).*\~\Z # (\A|/).*\.a\Z # (\A|/).*\.bak\Z # (\A|/).*\.BAK\Z # (\A|/).*\.elc\Z # (\A|/).*\.exe\Z # (\A|/).*\.gz\Z # (\A|/).*\.ln\Z # (\A|/).*\.o\Z # (\A|/).*\.obj\Z # (\A|/).*\.olb\Z # (\A|/).*\.old\Z # (\A|/).*\.orig\Z # (\A|/).*\.rej\Z # (\A|/).*\.so\Z # (\A|/).*\.Z\Z # (\A|/)\.del\-.*\Z # (\A|/)\.make\.state\Z # (\A|/)\.nse_depinfo\Z # (\A|/)core\Z # (\A|/)tags\Z # (\A|/)TAGS\Z # p "MANIFEST" 1260 1241684100 0100666 # p "lib/YAML.pm" 24651 1241624623 0100666 # c "t/dump-file-utf8.t" 0 1241683926 0100666 # p "t/meta-yml.t" 1007 1241683172 0100666 #### End of ApplyPatch data #### #### End of Patch kit [created: Thu May 7 10:17:05 2009] #### #### Patch checksum: 124 3789 51491 #### #### Checksum: 154 4839 8734 ####
Le Jeu. Mai. 07 04:36:12 2009, DOLMEN a écrit : Show quoted text
> Here is a fix, including test cases. > > Note that fixing this bug fixes also RT43764, a Module::Build bug > related to encoding problem in META.yml generation.
Anyone listing for reviewing and committing this 3 months-old patch?
Subject: Re: [rt.cpan.org #25434] [PATCH] Unicode/utf8 support missing
Date: Tue, 18 Aug 2009 00:46:56 +1000
To: bug-YAML [...] rt.cpan.org
From: Adam Kennedy <adamkennedybackup [...] gmail.com>
I can add a commit bit for you if you'd like to help by applying it. Adam K 2009/8/17 Olivier 'dolmen' Mengué via RT <bug-YAML@rt.cpan.org>: Show quoted text
>       Queue: YAML >  Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=25434 > > > Le Jeu. Mai. 07 04:36:12 2009, DOLMEN a écrit :
>> Here is a fix, including test cases. >> >> Note that fixing this bug fixes also RT43764, a Module::Build bug >> related to encoding problem in META.yml generation.
> > Anyone listing for reviewing and committing this 3 months-old patch? > >
Le Lun. Aoû. 17 10:47:14 2009, adam@ali.as a écrit : Show quoted text
> I can add a commit bit for you if you'd like to help by applying it. > > Adam K
Yes, thank you. Olivier.
Subject: Re: [rt.cpan.org #25434] [PATCH] Unicode/utf8 support missing
Date: Wed, 19 Aug 2009 20:31:56 +1000
To: bug-YAML [...] rt.cpan.org
From: Adam Kennedy <adamkennedybackup [...] gmail.com>
What is your CPAN login id? If you don't have one, what is the prefered email for you? Adam K 2009/8/18 Olivier 'dolmen' Mengué via RT <bug-YAML@rt.cpan.org>: Show quoted text
>       Queue: YAML >  Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=25434 > > > Le Lun. Aoû. 17 10:47:14 2009, adam@ali.as a écrit :
>> I can add a commit bit for you if you'd like to help by applying it. >> >> Adam K
> > Yes, thank you. > > Olivier. >
Le Mer. Aoû. 19 06:32:25 2009, adam@ali.as a écrit : Show quoted text
> What is your CPAN login id?
DOLMEN
Le Lun. Aoû. 17 10:47:14 2009, adam@ali.as a écrit : Show quoted text
> I can add a commit bit for you if you'd like to help by applying it. > > Adam K
Fixed in revision 8696+8697. http://fisheye2.atlassian.com/changelog/cpan/?cs=8696 http://fisheye2.atlassian.com/changelog/cpan/?cs=8697 Status could be set to 'stagnant' (waiting for a release).
Le Jeu. Aoû. 20 10:13:34 2009, DOLMEN a écrit : Show quoted text
Adam, could you review this 3-months old fix and make a new release? -- Olivier Mengué - http://o.mengue.free.fr/