Skip Menu |

This queue is for tickets about the Finance-InteractiveBrokers-SWIG CPAN distribution.

Report information
The Basics
Id: 96322
Status: resolved
Worked: 1 hour (60 min)
Priority: 0/
Queue: Finance-InteractiveBrokers-SWIG

People
Owner: INFIDEL [...] cpan.org
Requestors: gdg [...] zplane.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Upgrading perl from 5.18.2 to 5.20.0 results in 'Bizarre copy...' error
Date: Sat, 7 Jun 2014 19:14:40 -0600
To: Bugs in Finance-InteractiveBrokers-SWIG via RT <bug-Finance-InteractiveBrokers-SWIG [...] rt.cpan.org>
From: Glenn Golden <gdg [...] zplane.com>
Hello again Jason, Apologies for tapping your shoulder yet again, but this looks like a show stopper, at least for anyone using perl 5.20: After upgrading from perl 5.18.2 to 5.20.0, calling eConnect() consistently produces the following error: Bizarre copy of UNKNOWN in scalar assignment at /[blah blah]/Finance/InteractiveBrokers/SWIG.pm line 206. F:IB versions in use are API 0.04, SWIG 0.12. I'm sure this is related to the perl version upgrade, as it did not occur prior to the upgrade, only after; nothing in my code changed and I did not rebuild any of your modules. (The perl upgrade was done as a result of a general Arch repo synch that I did earlier today, so quite a few other packages got upgraded along with perl.) After doing some homework reading up on this error (though honestly not entirely understanding what I read) it seemed that one possibility is that 5.20.0 perl was built with options that are incompatible with the options used when building the module. So, out of superstition, I tried rebuilding both F::IB::API and F::IB::SWIG after the general system upgrade. No help. Downgrading back to 5.18.2 restored previous (normal) behavior, but I do not know (yet!) what else in my system may break as a result of using a now out of date perl, so this is only a temporary fix. Let me know what other info you may need, gdb-ing, etc. Glenn
I haven't had time to set up and to reproduce this yet, but from skimming, there appears to be an issue (possibly SWIG-related) due to an internal code change between perl 5.10 and perl 5.12: http://sourceforge.net/p/swig/patches/264/ More info: http://subversion.1072662.n5.nabble.com/PATCH-perl-bindings-Bizarre-copy-of-UNKNOWN-in-subroutine-td38024.html http://sourceforge.net/p/swig/patches/264/ Two things: * An integer argument is apparently being interpreted as a pointer in SWIG itself, which is reportedly patched in 2.04 (untested). * Since the 5.10->5.12 switch happened well before 5.18->5.20, I suspect there is also potential for library incompatibility if you do the following: ** Compile F::IB::SWIG with one version of Perl ** Change perl versions, and expect the libraries to work with a different version. Due to compilation arguments, and the extensive interrogation of perl (exe) settings I have to do during build time in this module, if your distro is compiling perls differently, this probably can't be helped in any reasonable fashion. I'll try to get a working test case and possible patch for this Real Soon Now(tm), but for the meantime, here's some suggestions: * Staying on perl 5.18.x * Recompiling F::IB::SWIG against the upgraded perl version. * Testing with SWIG 2.04 (also would need a recompile.) Doing this separately from the above would be more informative. If you find useful info from the above, please update this ticket. Also, 'perlbrew' might be useful for generating local binaries in a deterministic fashion (instead of relying on distro-built binaries), as well as running prod code on a known-to-be-compatible set of software versions, while testing version changes in a staging env. P.S. World Cup is starting, so a fix for this may take a bit...
Subject: Re: [rt.cpan.org #96322] Upgrading perl from 5.18.2 to 5.20.0 results in 'Bizarre copy...' error
Date: Tue, 10 Jun 2014 16:15:05 -0600
To: Jason McManus via RT <bug-Finance-InteractiveBrokers-SWIG [...] rt.cpan.org>
From: Glenn Golden <gdg [...] zplane.com>
Hi Jason, Jason McManus via RT <bug-Finance-InteractiveBrokers-SWIG@rt.cpan.org> [2014-06-10 10:17:08 -0400]: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=96322 > > > I haven't had time to set up and to reproduce this yet, but from skimming, > there appears to be an issue (possibly SWIG-related) due to an internal > code change between perl 5.10 and perl 5.12: > http://sourceforge.net/p/swig/patches/264/ > > [ ... ] >
That patch was applied to SWIG 2.0.4; I have 3.0.2, so suspect it is unrelated, unless its a regression. Show quoted text
> > * Since the 5.10->5.12 switch happened well before 5.18->5.20, I suspect > there is also potential for library incompatibility if you do the > following: > > ** Compile F::IB::SWIG with one version of Perl > ** Change perl versions, and expect the libraries to work with a > different version. Due to compilation arguments, and the extensive > interrogation of perl (exe) settings I have to do during build time > in this module, if your distro is compiling perls differently, > this probably can't be helped in any reasonable fashion. >
Yes, that was what I took away from my short background reading too, and I did try rebuilding both F::IB::SWIG and F::IB::API via CPAN under 5.20.0 before filing the ticket. No change in observed fault behavior. The fault was seen under perl 5.20.0 with the libraries built under both 5.18.2 and under 5.20.0. Show quoted text
> > I'll try to get a working test case and possible patch for this Real Soon > Now(tm) [ ...] > > If you find useful info from the above, please update this ticket.
See attached minimal example. Let me know if you're able to reproduce the problem using that. Was able to narrow it down to some sort of peculiar interaction with an open filehandle (!). See attached minimal example. Show quoted text
> > P.S. World Cup is starting, so a fix for this may take a bit... >
No problem, Jason, enjoy! Again, I really appreciate your responsiveness. I'm ok with downgrading to 5.18.2 so far, no problems seen against other perl-based tools. Regards, Glenn

Message body is not shown because sender requested not to inline it.

Shot in the dark here, as I can't reproduce this. Can you try unpacking a clean F::IB::SWIG 0.12 distro, dropping in the attached replacement Makefile.swig, building under perl 5.20.0, and seeing if the 'Bizarre copy' error recurs? If the drop-in copy doesn't help, can you attach: * your full 'perl -V' output for 5.18.x and 5.20.0 * the output of the "make" step of "perl Makefile.PL;make;make test" for both 5.18.x and 5.20.0 * the output of 'file IBAPI.so' (the unix file(1) command) for built libraries under both 5.18.x and 5.20.0 Thanks, -J
Subject: Makefile.swig
Download Makefile.swig
application/octet-stream 4.1k

Message body not shown because it is not plain text.

Subject: Re: [rt.cpan.org #96322] Upgrading perl from 5.18.2 to 5.20.0 results in 'Bizarre copy...' error
Date: Sun, 15 Jun 2014 08:15:26 -0600
To: Jason McManus via RT <bug-Finance-InteractiveBrokers-SWIG [...] rt.cpan.org>
From: Glenn Golden <gdg [...] zplane.com>
Jason McManus via RT <bug-Finance-InteractiveBrokers-SWIG@rt.cpan.org> [2014-06-15 02:34:20 -0400]: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=96322 > > > Shot in the dark here, as I can't reproduce this. > > Can you try unpacking a clean F::IB::SWIG 0.12 distro, dropping in the attached replacement Makefile.swig, building under perl 5.20.0, and seeing if the 'Bizarre copy' error recurs? > > If the drop-in copy doesn't help, can you attach: > * your full 'perl -V' output for 5.18.x and 5.20.0 > * the output of the "make" step of "perl Makefile.PL;make;make test" for both 5.18.x and 5.20.0 > * the output of 'file IBAPI.so' (the unix file(1) command) for built libraries under both 5.18.x and 5.20.0 >
First of all, thanks for taking a break from World Cup to work on this! :) Much appreciated. Before doing the above experiments though, I need to ask for a little quick help to ensure that when I report the results back to you I wind up giving you reliable information and not getting myself confused and wasting your (and my) time. Specifically, I want to perform these experiments in a way which is extremely anal with regard to precisely which directory SWIG.pm (and its associated IBAPI.so) is being "use"d from when I execute the minimal example under the various experimental combinations (original vs. modified Makefile.swig, and the two perl versions.) With all those variations, there are a lot of ways to make mistakes. I added some barfage at the top of the example to report the paths from @INC and also to report the value of $INC{[blah blah]/SWIG.pm} so I can see unambiguously which one is being use-ed, that's the easy part. But what I'd also like to do -- and this is where I need some help -- is completely segregate the install trees for SWIG based on the experimental condition, i.e. one for 5.18.2, one for 5.20.0, and then perhaps subtrees (or separate trees) for switching between the original Makefile.swig and the modified one that you supplied. Doing it this way will be less error prone than overwriting the install directory contents each time I switch perl versions. So my question is this: When doing the "make install", how can I specify the *full path* of the installation directory, so that when running the minimal example for a particular experimental case, I can simply prepend the desired SWIG installation directory at the front of @INC in order to get it to use the intended SWIG.pm version?
Subject: Re: [rt.cpan.org #96322] Upgrading perl from 5.18.2 to 5.20.0 results in 'Bizarre copy...' error
Date: Sun, 15 Jun 2014 11:21:20 -0600
To: Jason McManus via RT <bug-Finance-InteractiveBrokers-SWIG [...] rt.cpan.org>
From: Glenn Golden <gdg [...] zplane.com>
Hi Jason, Show quoted text
Update: After playing with your updated Makefile.swig and finding that it seemed to "fix" the problem, I went back and tried to reproduce the fault behavior using the original Makefile.swig, just to be sure the difference between the original and updated Makefile.swig was really responsible for the fix. But after manually rebuilding/reinstalling SWIG using the original Makefile.swig... it wouldn't fail! So the Makefile.swig actually has nothing to do with the problem. After a few additional hours of fiddling around, here's what I've concluded as to the real source of the problem: When I had originally rebuilt SWIG after upgrading my perl version to 5.20.0 (prior to filing the ticket) I used CPAN, as follows: 1. Removed all extant ~/.cpan/build/F::IB::SWIG-0.12-* dirs, just to be safe. 2. Then, from CPAN commandline: get F::IB::SWIG # Completed normally make F::IB::SWIG # Completed normally test F::IB::SWIG # Completed normally, all tests passed install F::IB::SWIG # Reports "F::IB::SWIG is up to date (0.12)" In the "install" step, I assumed that the message "SWIG is up to date" meant that after the "install" just performed, SWIG was "now up to date". But evidently what it really meant was that it [mistakenly] determined that the installed SWIG components were *already* up to date (relative to version 0.12) and therefore CPAN would not perform any of the installation steps which would have [correctly] overwritten the existing SWIG.pm and IBAPI.so in the installation tree. This explains why my app failed under 5.20.0, even though I had rebuilt and "reinstalled" SWIG after the upgrade to 5.20.0. The CPAN "install" had done nothing, so my app was still use-ing the versions built months ago under 5.18.2. One could guess that the underlying problem is that either CPAN somehow tried (and failed) to outsmart the modification-time based dependencies in the F::IB Makefile, or the Makefile dependencies themselves are faulty in some way. It seems to me that it cannot be right that that CPAN "install" command should wind up as a no-op under the condition that it had just rebuilt IBAPI.so. It's modification time relative to the installed version should have triggered an unconditional overwrite. Btw, I reproduced the above CPAN sequence several times from scratch, just before sending this message, just to be sure I wasn't imagining it or misinterpreting what had gone on. But the no-op behavior was identical each time, as verified by the extant SWIG.pm and IBAPI.so files in the installation tree being untouched after the "install" command (no change to their creation or modification times). So I am quite sure this is the source of the problem. In case you want to try to reproduce this behavior from within CPAN, I attached my CPAN config file ("MyConfig.pm"), in case it might somehow be implicated. Also attached is a script ("setlocperl") which I source prior to running CPAN, to set up the environment appropriately. It is somewhat non-standard in the sense that it sets things up for user-based installation tree (i.e. installs into /home/XXX/locperl) rather than system-based paths, so possibly this could also be implicated. Anyway, let me know if you need any additional info in tracking down the install issue, and once again thanks for your time on this. Regards, Glenn

Message body is not shown because sender requested not to inline it.

Message body is not shown because sender requested not to inline it.

Ah, this makes sense; so perl 5.20.0 was running against the library built by 5.18.2, leading to binary incompatibility. The 'cpan' executable isn't something I can control, and it only does a rudimentary $Module::VERSION check against the installed primary perl module (F::IB::SWIG.pm in this case, which would still report 0.12 in both cases), instead of a full 'make'-like mtime check. It has a "force install" command though, which would overwrite the installed modules. Incidentally, there's a few different tools for localized perl installations you may want to look into: * local::lib (installs modules to a user directory) * perlbrew (allows multiple perl version installations outside of system perl) * App::cpanminus (a 'cpan' replacement, tends to be a bit faster ime) I still may release a 0.13 F::IB::SWIG with the replacement Makefile.swig in a bit, as I noticed during this the FILE_OFFSET_BITS and LARGEFILE_SOURCE defines were not being carried forward when building the IBAPI-supplied code, but for now, it looks like this is sorted. Cheers.
Subject: Re: [rt.cpan.org #96322] Upgrading perl from 5.18.2 to 5.20.0 results in 'Bizarre copy...' error
Date: Mon, 16 Jun 2014 17:20:15 -0600
To: Jason McManus via RT <bug-Finance-InteractiveBrokers-SWIG [...] rt.cpan.org>
From: Glenn Golden <gdg [...] zplane.com>
Jason McManus via RT <bug-Finance-InteractiveBrokers-SWIG@rt.cpan.org> [2014-06-15 19:00:48 -0400]: Show quoted text
Show quoted text
> > Ah, this makes sense; so perl 5.20.0 was running against the library built > by 5.18.2, leading to binary incompatibility. >
Yep. Show quoted text
> > The 'cpan' executable isn't something I can control, and it only does a > rudimentary $Module::VERSION check against the installed primary perl > module (F::IB::SWIG.pm in this case, which would still report 0.12 in both > cases), instead of a full 'make'-like mtime check. >
I realize cpan isn't your responsibility at all, I just mentioned about it and supplied the config files in case you wanted to investigate whether the no-op-ness of the cpan install was somehow related to your Makefile itself (as I had assumed was the case). But from what you say above, cpan doesn't simply run "make install" using the module makefile, it just looks at the version number of the installed module and makes a "lite" (and often wrong) guess based only on that as to whether to run "make install" or not. Show quoted text
> > It has a "force install" command though, which would overwrite the > installed modules. >
Yep, ironically, I knew about that but the reported message about "is up to date" fooled me into thinking that the install had in fact occurred, so I didn't even think about trying to force it. I assumed (wrongly) that the cpan "install" command had executed "make install" on the module Makefile and was simply reporting (after completion) that it was now up to date. Show quoted text
> > Incidentally, there's a few different tools for localized perl installations > you may want to look into: > * local::lib (installs modules to a user directory) >
This is that I used to do the install of cpan itself and then to create a user directory for the install tree. Seems to work pretty well. Show quoted text
> > * perlbrew (allows multiple perl version installations outside of system perl) >
Ja, you mentioned this earlier, I'll check it out sometime. Could definitely come in handy when trying to diagnose problems like this one. Show quoted text
> > * App::cpanminus (a 'cpan' replacement, tends to be a bit faster ime) >
Ah, that one I have to certainly check out because CPAN has been a constant source of annoyance for me. Seems to me it introduces more complications than simply doing the build by hand, and it's not particularly well documented either, imo. So perhaps the "minus" version will be a welcome reduction in fussiness, thanks for mentioning it. Show quoted text
> I still may release a 0.13 F::IB::SWIG with the replacement Makefile.swig in > a bit, as I noticed during this the FILE_OFFSET_BITS and LARGEFILE_SOURCE > defines were not being carried forward when building the IBAPI-supplied code, > but for now, it looks like this is sorted.
Indeed, and thanks again for your time, Jason. Regards, Glenn