Skip Menu |

This queue is for tickets about the Text-Tokenizer CPAN distribution.

Report information
The Basics
Id: 72326
Status: resolved
Priority: 0/
Queue: Text-Tokenizer

People
Owner: Nobody in particular
Requestors: Bill.Costa [...] unh.edu
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Build error on Mac OS for version 0.4.3 of Text::Tokenizer
Date: Wed, 9 Nov 2011 16:28:57 -0500 (EST)
To: bug-Text-Tokenizer [...] rt.cpan.org
From: Bill.Costa [...] unh.edu
This module looks like exactly what I need, so I was disappointed when I couldn't build it. Distro Name and Version: Text::Tokenizer version 0.4.3 $ perl -v This is perl 5, version 12, subversion 3 (v5.12.3) built for darwin-multi-2level $ uname -a Darwin dolphin.unh.edu 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 - - - - - - - - - - - - - - $ make $ make test PERL_DL_NONLAZY=1 /opt/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/1.t .. ok t/2.t .. 1/10 # Failed test 'Tokenizer exists' # at t/2.t line 12. # Failed test 'Tokenizer switch' # at t/2.t line 13. # Failed test 'Tokenizer delete' # at t/2.t line 43. # Looks like you failed 3 tests of 10. t/2.t .. Dubious, test returned 3 (wstat 768, 0x300) Failed 3/10 subtests t/3.t .. 1/10 # Failed test 'Tokenizer exists' # at t/3.t line 12. # Failed test 'Tokenizer switch' # at t/3.t line 13. # Failed test 'Tokenizer delete' # at t/3.t line 45. # Looks like you failed 3 tests of 10. t/3.t .. Dubious, test returned 3 (wstat 768, 0x300) Failed 3/10 subtests Test Summary Report ------------------- t/2.t (Wstat: 768 Tests: 10 Failed: 3) Failed tests: 4-5, 8 Non-zero exit status: 3 t/3.t (Wstat: 768 Tests: 10 Failed: 3) Failed tests: 4-5, 8 Non-zero exit status: 3 Files=3, Tests=22, 0 wallclock secs ( 0.03 usr 0.01 sys + 0.09 cusr 0.02 csys = 0.15 CPU) Result: FAIL Failed 2/3 test programs. 6/22 subtests failed. make: *** [test_dynamic] Error 3 $ - - - - - - - - - - - - - - Tried: $ flex -V flex 2.5.35 $ make -f Makefile.flex rm -f lex.tokenizer_yy.o flex -Cfar --nounistd tokenizer.c.flex rm -f lex.tokenizer_yy.c.orig $ but the make/make test still didn't work, failing as above. I looked at the first failed test and am at a loss at how I could even start to debug this given that it is failing to return a tokenizer without any sort of error message. I'm motivated to get this to work, so let me know what I can do to help debug this. ...BC -- =====================================[ Bill.Costa@unh.edu ]== Bill Costa 1 Leavitt Lane UNH IT -- 1st Floor University of New Hampshire Durham, NH 03824 USA Voice: +1-603-862-3056 No good deed... Goes unpunished. ===========================[ http://pubpages.unh.edu/~wfc ]==
Hi, I'm don't know what is the problem, but it fails only on BSD-flavor OS-es, every other (including crappy old UNIXes have no problem). The problem seems to be in a way Text-Tokenizer is generating unique tokenizer ID - but I'm using pointer address for this: You can see it in tokenizer.c.flex: /* * setup default tokenizer */ static tok_id tokenizer_init(FILE *f) { .... tokb->id = f; .... return TOKEN_ID(tokb->id); } No idea what wrong about that, and I have no BSD-flavor OS to test it. I can try to get one, but it may take some time...
Subject: Re: [rt.cpan.org #72326] Build error on Mac OS for version 0.4.3 of Text::Tokenizer
Date: Thu, 10 Nov 2011 16:51:10 -0500 (EST)
To: Samuel Behan via RT <bug-Text-Tokenizer [...] rt.cpan.org>
From: Bill.Costa [...] unh.edu
Samuel, Thanks for the speedy reply... Show quoted text
> No idea what wrong about that, and I have no BSD-flavor OS to > test it. I can try to get one, but it may take some time...
If I setup an account on my desktop Mac for you, and dump the distribution there, would you be willing to look at it via SSH? I'll be back in the office on Monday and can set things up then. I'll need a secure way of getting the account name and password to you. A phone call to you would probably be the easiest. ...BC -- =====================================[ Bill.Costa@unh.edu ]== Bill Costa 1 Leavitt Lane UNH IT -- 1st Floor University of New Hampshire Durham, NH 03824 USA Voice: +1-603-862-3056 No good deed... Goes unpunished. ===========================[ http://pubpages.unh.edu/~wfc ]==
Hi, i've reviewed some CPAN tester reports and found a root of this problem. It seems to appear only for specific combinations of architecture, OS and perl version and has smth. to do with differently defined type of void* within PERL and OS. However I've made a change that should fix this and released version 0.4.4 of Text::Tokenizer. You can download it from http://devel.dob.sk/Text-Tokenizer/Text-Tokenizer-0.4.4.tar.gz, or from CPAN it should be there in a while. Please let me know if it works :)
Subject: Re: [rt.cpan.org #72326] Build error on Mac OS for version 0.4.3 of Text::Tokenizer
Date: Fri, 11 Nov 2011 10:25:59 -0500 (EST)
To: Samuel Behan via RT <bug-Text-Tokenizer [...] rt.cpan.org>
From: Bill.Costa [...] unh.edu
Samuel, Show quoted text
> Hi, i've reviewed some CPAN tester reports and found a root of > this problem.
Excellent! Show quoted text
> Please let me know if it works :)
Count on it. Looking forward to using your module. I have a need for exactly this capability and it saves me from digging out and translating a similiar program I wrote for as part of a compiler for my masters thesis -- 21 years ago -- in Pascal! :-) Thanks for looking into this and fixing it so quickly. ...BC -- =====================================[ Bill.Costa@unh.edu ]== Bill Costa 1 Leavitt Lane UNH IT -- 1st Floor University of New Hampshire Durham, NH 03824 USA Voice: +1-603-862-3056 No good deed... Goes unpunished. ===========================[ http://pubpages.unh.edu/~wfc ]==
resolved in version 0.4.5