Bug #21091 for Test-Simple: UTF-8 Support

Fri Oct 21 07:35:12 2005 TELS [...] cpan.org - Ticket #15187: Ticket created

Subject:

Test::Builder::is() trips over utf8 strings in fail reports

#!/usr/bin/perl -w use Test::More tests => 1; sub _u8 { # Converts UTF-16 codepoints in hex (like "2500") into utf-8 my ($utf16) = @_; require Encode; Encode::decode('UTF-16BE', pack("H" . length($utf16), $utf16)); } binmode STDOUT, ':utf8'; # this does not help binmode STDERR, ':utf8'; # neither does this my $a = _u8('2500'); my $b = _u8('2501'); print STDERR "# comparing $a to $b\n"; is ($a, $b, "$a eq $b ?"); __END__ The above file will output the following on a console with Unicode support: 1..1 # comparing ─ to ━ Wide character in print at /usr/local/lib/perl5/5.8.6/Test/Builder.pm line 1182. not ok 1 - ─ eq ━ ? Wide character in print at /usr/local/lib/perl5/5.8.6/Test/Builder.pm line 1199. # Failed test '─ eq ━ ?' # in utf8.pl at line 30. Wide character in print at /usr/local/lib/perl5/5.8.6/Test/Builder.pm line 1199. # got: '─' # expected: '━' # Looks like you failed 1 test of 1. Best wishes, Tels

Sat Oct 22 03:01:18 2005 schwern [...] pobox.com - Ticket #15187: Correspondence added

Date:	Sat, 22 Oct 2005 00:00:49 -0700
From:	Michael G Schwern <schwern [...] pobox.com>
To:	via RT <bug-Test-Simple [...] rt.cpan.org>
CC:	undisclosed-recipients: ;
Subject:	Re: [cpan #15187] Test::Builder::is() trips over utf8 strings in fail reports
RT-Send-Cc:

On Fri, Oct 21, 2005 at 07:35:12AM -0400, via RT wrote: Show quoted text

> The above file will output the following on a console with Unicode support:

I know pretty much nothing about Unicode. Suggestions? -- Michael G Schwern schwern@pobox.com http://www.pobox.com/~schwern Just call me 'Moron Sugar'. http://www.somethingpositive.net/sp05182002.shtml

Sun Aug 20 11:22:58 2006 HDM [...] cpan.org - Ticket created

Subject:

UTF-8 Support

What would you think about some kind of "utf8" import that lets you enable UTF8 support on the filehandles that Test::More uses? At present, I'm having to do things like this: use Test::More 'no_plan'; { my $Test = Test::Builder->new; binmode( $Test->output, ":utf8" ); binmode( $Test->failure_output, ":utf8" ); binmode( $Test->todo_output, ":utf8" ); } Ideally, I'd like to be able to say use Test::More -utf8, 'no_plan'; Or similar.

Sun Aug 20 11:24:02 2006 HDM [...] cpan.org - Broken in 0.62 added

Sun Aug 20 11:24:02 2006 HDM [...] cpan.org - Fixed in 0.62 deleted

Tue May 08 17:14:03 2007 burak [...] cpan.org - Ticket #27015: Ticket created

Subject:

[Test::More] Wide character in print warnings

Perl 5.8.8 & Windows XP Pro SP2 Test code: ----------------------------------------- use 5.008; use strict; use warnings; use Test::More tests => 1; my $uni = "\x{11e}"; ok( $uni eq $uni, "Testing $uni" ); printf "Test::More is v%s\n", $Test::More::VERSION; ----------------------------------------- Output: ----------------------------------------- 1..1 Wide character in print at C:/Perl/lib/Test/Builder.pm line 1245. ok 1 - Testing Ğ Test::More is v0.7 ----------------------------------------- This warning can be by-passed with this awkward code: binmode Test::More->builder->output, ':utf8'; Can there be a nicer interface for this? perhaps an import() parameter?

Tue Nov 06 09:15:28 2007 cornelius.howl [...] gmail.com - Ticket #30518: Ticket created

Subject:

Wide character in print

when i print chinese , it warns: WARN - Wide character in print at /usr/local/share/perl/5.8.8/Test/Builder.pm line 1247. could you add something like: binmode $fh,":utf8"; thanks. :)

Sat Feb 23 23:19:01 2008 mschwern [...] cpan.org - Ticket #15187: Merged into ticket #21091

Sat Feb 23 23:19:01 2008 mschwern [...] cpan.org - Merged into ticket #21091

Sat Feb 23 23:19:10 2008 mschwern [...] cpan.org - Ticket #27015: Merged into ticket #21091

Sat Feb 23 23:19:10 2008 mschwern [...] cpan.org - Merged into ticket #21091

Sat Feb 23 23:19:18 2008 mschwern [...] cpan.org - Ticket #30518: Merged into ticket #21091

Sat Feb 23 23:19:18 2008 mschwern [...] cpan.org - Merged into ticket #21091

Sat Feb 23 23:43:43 2008 mschwern [...] cpan.org - Correspondence added

0.76_01 will use :utf8 on the default output handles. That should quiet any warnings.

Sat Feb 23 23:43:45 2008 The RT System itself - Status changed from 'new' to 'open'

Sat Feb 23 23:43:45 2008 mschwern [...] cpan.org - Status changed from 'open' to 'resolved'

Fri Mar 07 13:32:14 2008 jdhedden [...] cpan.org - Ticket #33884: Ticket created

Subject:

t/utf8.t failures under 5.8.8 on Solaris

'make test' fails with the following: t/useing..........................ok t/utf8............................Cannot find encoding "646" at /home/jhedden/local/lib/perl5/5.8.8/open.pm line 125. Cannot find encoding "646" at /home/jhedden/local/lib/perl5/5.8.8/open.pm line 133. Out of memory! Out of memory! t/utf8............................ Dubious, test returned 1 (wstat 256, 0x100) No subtests run Test Summary Report ------------------- t/utf8.t (Wstat: 256 Tests: 0 Failed: 0) Non-zero exit status: 1 Parse errors: No plan found in TAP output Files=73, Tests=521, 102 wallclock secs ( 1.09 usr 0.90 sys + 48.10 cusr 45.33 csys = 95.42 CPU) Result: FAIL Running the test directly, produces: Show quoted text

> perl -Mblib t/utf8.t

Cannot find encoding "646" at /home/jhedden/local/lib/perl5/5.8.8/open.pm line 125. Cannot find encoding "646" at /home/jhedden/local/lib/perl5/5.8.8/open.pm line 133. Segmentation Fault Show quoted text

> perl -V

Summary of my perl5 (revision 5 version 8 subversion 8) configuration: Platform: osname=solaris, osvers=2.9, archname=sun4-solaris-thread-multi-64int uname='sunos sundev3 5.9 generic_112233-12 sun4u sparc sunw,sun-fire ' config_args='-de -Dusethreads=yes -Duse64bitint=yes -Dusedevel=yes' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemultiplicity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=define use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='/opt/SUNWspro/bin/cc', ccflags ='-D_REENTRANT -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O', cppflags='-D_REENTRANT -I/usr/local/include' ccversion='Sun WorkShop 6 update 1 C 5.2 2000/09/11', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=87654321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='/opt/SUNWspro/bin/cc', ldflags =' -L/usr/lib -L/usr/ccs/lib -L/opt/SUNWspro/WS6U1/lib -L/usr/local/lib ' libpth=/usr/lib /usr/ccs/lib /opt/SUNWspro/WS6U1/lib /usr/local/lib libs=-lsocket -lnsl -lgdbm -ldl -lm -lpthread -lc perllibs=-lsocket -lnsl -ldl -lm -lpthread -lc libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' ' cccdlflags='-KPIC', lddlflags='-G -L/usr/lib -L/usr/ccs/lib -L/opt/SUNWspro/WS6U1/lib -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API Built under solaris Compiled at May 30 2006 14:17:36 %ENV: PERL5LIB="/bb/util/common/perlmod/lib/site_perl" PERLIO="perlio" @INC: /bb/util/common/perlmod/lib/site_perl /home/jhedden/local/lib/perl5/5.8.8/sun4-solaris-thread-multi-64int /home/jhedden/local/lib/perl5/5.8.8 /home/jhedden/local/lib/perl5/site_perl/5.8.8/sun4-solaris-thread-multi-64int /home/jhedden/local/lib/perl5/site_perl/5.8.8 /home/jhedden/local/lib/perl5/site_perl .

Wed Mar 12 17:43:59 2008 david.buice [...] eds.com - Ticket #34035: Ticket created

Subject:	t/utf8.t failures under 5.8.4 on Solaris
Date:	Wed, 12 Mar 2008 15:11:13 -0400
To:	<bug-Test-Simple [...] rt.cpan.org>
From:	"Buice, David R" <david.buice [...] eds.com>

'make test' fails with the following: t/useing........................ok t/utf8..........................Cannot find encoding "646" at /usr/perl5/5.8.4/lib/open.pm line 126. Cannot find encoding "646" at /usr/perl5/5.8.4/lib/open.pm line 134. t/utf8..........................dubious Test Summary Report ---------------------- Test returned status 0 (wstat 139, 0x8b) test program seems to have generated a core Failed Test Stat Wstat Total Fail Failed List of Failed ------------------------------------------------------------------------ ------- t/utf8.t 0 139 ?? ?? % ?? 5 tests and 13 subtests skipped. Failed 1/73 test scripts, 98.63% okay. 0/515 subtests failed, 100.00% okay. *** Error code 2 make: Fatal error: Command failed for target `test_dynamic' Running the test directly produces: perl -Mblib t/utf8.t Cannot find encoding "646" at /usr/perl5/5.8.4/lib/open.pm line 126. Cannot find encoding "646" at /usr/perl5/5.8.4/lib/open.pm line 134. Segmentation Fault(coredump) perl -V Summary of my perl5 (revision 5 version 8 subversion 4) configuration: Platform: osname=solaris, osvers=2.10, archname=sun4-solaris-64int uname='sunos localhost 5.10 sun4u sparc SUNW,Ultra-2' config_args='' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=define use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -xarch=v8 -D_TS_ERRNO', optimize='-xO3 -xspace -xildoff', cppflags='' ccversion='Sun WorkShop', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=87654321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags ='' libpth=/lib /usr/lib /usr/ccs/lib libs=-lsocket -lnsl -ldl -lm -lc perllibs=-lsocket -lnsl -ldl -lm -lc libc=/lib/libc.so, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-R /usr/perl5/5.8.4/lib/sun4-solaris-64int/CORE' cccdlflags='-KPIC', lddlflags='-G' Characteristics of this binary (from libperl): Compile-time options: USE_64_BIT_INT USE_LARGE_FILES Locally applied patches: 22667 The optree builder was looping when constructing the ops ... 22715 Upgrade to FileCache 1.04 22733 Missing copyright in the README. 22746 fix a coredump caused by rv2gv not fully converting a PV ... 22755 Fix 29149 - another UTF8 cache bug hit by substr. 22774 [perl #28938] split could leave an array without ... 22775 [perl #29127] scalar delete of empty slice returned garbage 22776 [perl #28986] perl -e "open m" crashes Perl 22777 add test for change #22776 ("open m" crashes Perl) 22778 add test for change #22746 ([perl #29102] Crash on assign ... 22781 [perl #29340] Bizarre copy of ARRAY make sure a pad op's ... 22796 [perl #29346] Double warning for int(undef) and abs(undef) ... 22818 BOM-marked and (BOMless) UTF-16 scripts not working 22823 [perl #29581] glob() misses a lot of matches 22827 Smoke [5.9.2] 22818 FAIL(F) MSWin32 WinXP/.Net SP1 (x86/1 cpu) 22830 [perl #29637] Thread creation time is hypersensitive 22831 improve hashing algorithm for ptr tables in perl_clone: ... 22839 [perl #29790] Optimization busted: '@a = "b", sort @a' ... 22850 [PATCH] 'perl -v' fails if local_patches contains code snippets 22852 TEST needs to ignore SCM files 22886 Pod::Find should ignore SCM files and dirs 22888 Remove redundant %SIG assignments from FileCache 23006 [perl #30509] use encoding and "eq" cause memory leak 23074 Segfault using HTML::Entities 23106 Numeric comparison operators mustn't compare addresses of ... 23320 [perl #30066] Memory leak in nested shared data structures ... 23321 [perl #31459] Bug in read() SPRINTF0 - fixes for sprintf formatting issues - CVE-2005-3962 Built under solaris Compiled at Feb 13 2006 05:12:02 @INC: /usr/perl5/5.8.4/lib/sun4-solaris-64int /usr/perl5/5.8.4/lib /usr/perl5/site_perl/5.8.4/sun4-solaris-64int /usr/perl5/site_perl/5.8.4 /usr/perl5/site_perl /usr/perl5/vendor_perl/5.8.4/sun4-solaris-64int /usr/perl5/vendor_perl/5.8.4 /usr/perl5/vendor_perl . Regards, David Buice ITIL Foundations Certified EDS - PSIC 7000 Chicago Road Warren, MI 48092 ( Phone:+1-586-575-4525 + mailto:david.buice@eds.com cell: 248-249-9122

Fri Jun 20 12:18:02 2008 mschwern [...] cpan.org - Ticket #34035: Severity Wishlist added

Fri Jun 20 12:18:15 2008 mschwern [...] cpan.org - Ticket #33884: Merged into ticket #21091

Fri Jun 20 12:18:15 2008 mschwern [...] cpan.org - Ticket #34035: Merged into ticket #21091

Sun Aug 10 06:08:57 2008 mschwern [...] cpan.org - Status changed from 'resolved' to 'open'

Sun Aug 10 06:11:59 2008 mschwern [...] cpan.org - Correspondence added

Sorry, it didn't work out. The necessary IO layer support in Perl is busted on a number of platforms.

Sat May 30 15:25:18 2009 schwern [...] pobox.com - Ticket #46542: Ticket created

Subject:	[Fwd: copy_io_layers in Test::More]
Date:	Sat, 30 May 2009 12:24:46 -0700
To:	bug-Test-Simple [...] rt.cpan.org
From:	Michael G Schwern <schwern [...] pobox.com>

Show quoted text

-------- Original Message -------- Subject: copy_io_layers in Test::More Date: Mon, 20 Apr 2009 06:58:33 -0400 From: David Golden <xdaveg@gmail.com> To: Michael G Schwern <schwern@pobox.com> I see the intent to copy IO layers, but it's commented out. Having just wrestled with IO layer copying for Capture::Tiny, I also see that the code won't work. Here's what I've figured out is required to copy layers -- at least in 5.10.0: sub _apply_layers { my ($fh, @layers) = @_; my %seen; my @unique = grep { $_ ne 'unix' and !$seen{$_}++ } @layers; binmode($fh, join(":", "", "raw", @unique)); } This will take layers as given by PerlIO::get_layers(STDOUT) and apply them to a new file handle. Stripping unix and duplicates is apparently necessary. Hope that helps if it's still on your todo list. I'd also welcome an option in Test::Builder to turn utf8 on/off selectively, not just when Test::Builder loads. Otherwise, I'm going to wind up doing this: my $saved_layers; BEGIN { $saved_layers = _save_layers(); binmode(STDOUT,":utf8"); binmode(STDERR,":utf8") } use Test::More BEGIN { _restore_layers($saved_layers) } I need utf8 to be off during most testing, but I turn it on for a few tests then turn it back off again. During those tests, I want Test::Builder to give me utf8 diagnostics. What do you think? -- David -- If you want the truth to stand clear before you, never be for or against. The struggle between "for" and "against" is the mind's worst disease. -- Sent-ts'an

Sat Aug 28 00:28:56 2010 mschwern [...] cpan.org - Ticket #46542: Requestor xdaveg@gmail.com added

Sat Aug 28 01:01:04 2010 mschwern [...] cpan.org - Ticket #46542: Correspondence added

Subject:

Fixing copy_io_layers in Test::More

Let me dig up the details of what the problems were last time I tried this... Here's one report: http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-03/msg00278.html And another: http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-03/msg00355.html 'use open ":std", ":locale"' is not safe on weird platforms. But Test::Builder only needs that for testing. And we only *really* care about utf8. So I can remove open.pm from the test and just use binmode() directly. I'll try that and release it as an alpha and see what sort of havoc it causes. Also, a method to apply filehandle disciplines to all the output handles at once would be appreciated.

Sat Aug 28 01:01:05 2010 The RT System itself - Ticket #46542: Status changed from 'new' to 'open'

Sat Aug 28 01:03:50 2010 mschwern [...] cpan.org - Ticket #46542: Correspondence added

I've pushed this in as 401fa01bc23c23f79ed0e97dfa7ea79422b3f5a8. Works on my machine, but it will need to hit a lot of others.

Sat Aug 28 01:53:40 2010 mschwern [...] cpan.org - Ticket #46542: Merged into ticket #21091

Sat Aug 28 01:53:40 2010 mschwern [...] cpan.org - Merged into ticket #21091

Sat Aug 28 01:54:20 2010 mschwern [...] cpan.org - Ticket #34035: Merged into ticket #21091

Sat Aug 28 01:54:20 2010 mschwern [...] cpan.org - Merged into ticket #21091

Fri Oct 28 15:48:14 2016 exodist7 [...] gmail.com - Correspondence added

Thanks to the Test2 change UTF8 support is now fairly trivial in a couple ways. I am closing this ticket. If you have further utf8 issues please open tickets for them on github.

Fri Oct 28 15:48:20 2016 exodist7 [...] gmail.com - Status changed from 'open' to 'resolved'

Bug #21091 for Test-Simple: UTF-8 Support

Preferred bug tracker