Bug #88097 for Finance-InteractiveBrokers-SWIG: Using with a GUI

Sun Aug 25 05:45:01 2013 richie765 [...] gmail.com - Ticket created

Subject:	Using with a GUI
Date:	Sun, 25 Aug 2013 11:44:48 +0200
To:	bug-Finance-InteractiveBrokers-SWIG [...] rt.cpan.org
From:	Richard <richie765 [...] gmail.com>

I've created a simple wxPerl app to do some simple interaction with IB. I can't figure out how to make it work. I've created an IDLE method to periodically call processMessages. But this locks the GUI for periods of time. Furthermore I keep getting the following error. Server Error: ReqID -1; code 509: Exception caught while reading socket - No such file or directory I tried many things to get rid of it but I can't. This error code isn't documented anywhere. I've looked at the source of processMessages but I don't understand it's function. I've also looked at the IBAPI GUI examples for C++ and Java, and it seems they don't call anything similar like "processMessages", the event handler just gets called without any such thing. What is it needed in Perl and what does it do exactly? I've also tried to fork and run the message loop in its own process, but that just results in errors. Do you have any suggestions how I could make this work? I can give more detailed information if needed.

Sun Aug 25 06:22:04 2013 INFIDEL [...] cpan.org - Correspondence added

Hi there. Show quoted text

> I've created an IDLE method to periodically call processMessages. But this > locks the GUI for periods of time.

That's sort of a known issue. I'm sharing the select() timeout to poll for messages with the ping timeout value in IBAPI.cxx. There is a workaround someone else came up for it here: https://rt.cpan.org/Ticket/Display.html?id=79926 I believe he modified the select() timeout on line 128 of IBAPI.cxx with something shorter. My use case (the one it was designed for) is that it runs as a daemon that writes to an intermediary FIFO queue, and then the queue is processed via other apps; the latency/lag doesn't matter in this situation, as it only does the poll/write loop. Show quoted text

> I've looked at the source of processMessages but I don't understand it's > function. I've also looked at the IBAPI GUI examples for C++ and Java, and > it seems they don't call anything similar like "processMessages", the event > handler just gets called without any such thing. What is it needed in Perl > and what does it do exactly?

It's not in the API itself, but it's a shrunken version of the call in their CPP PosixTestClient distributed with their app. It's a fairly standard event loop processor; it polls the socket for incoming messages with a timeout; but as you noticed above, it blocks for a possibly undesirably long period of time, if used directly in an interactive program. Show quoted text

> Furthermore I keep getting the following error. > > Server Error: ReqID -1; code 509: Exception caught while reading socket - > No such file or directory > > I tried many things to get rid of it but I can't. This error code isn't > documented anywhere.

For this one, I'm not sure offhand, and I've had to see what you're doing. This is the error message reference: https://www.interactivebrokers.com/en/software/api/apiguide/tables/api_message_codes.htm That error is generated in MyEventHandler.pm by the 'error()' sub, and it is called as a result of receiving a message from the server (well, the TWS endpoint or daemon), so it is something IB is sending to your client. If you're familiar with strace and/or wireshark, those tools would likely be beneficial in troubleshooting this issue, so you can see roughly what calls are being made and what is being communicated over the network. Furthermore, the IB daemon or TWS have their own logs that might also help. Are you passing a valid Request ID to the API? Show quoted text

> Do you have any suggestions how I could make this work? I can give more > detailed information if needed.

I might be able to take a look at a simple test case. If you can provide a short example of this, it's possible that I can recommend something for better architecting your event loop; but for the error above, it is unlikely I'd be able to provide more than the recommendations above.

Sun Aug 25 06:22:04 2013 The RT System itself - Status changed from 'new' to 'open'

Sun Aug 25 06:25:32 2013 INFIDEL [...] cpan.org - Taken

Sun Aug 25 10:55:47 2013 richie765 [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #88097] Using with a GUI
Date:	Sun, 25 Aug 2013 16:55:32 +0200
To:	bug-Finance-InteractiveBrokers-SWIG [...] rt.cpan.org
From:	Richard <richie765 [...] gmail.com>

Thanks for the fast response! That's sort of a known issue. I'm sharing the select() timeout to poll for Show quoted text

> messages with the ping timeout value in IBAPI.cxx. There is a workaround > someone else came up for it here: > https://rt.cpan.org/Ticket/Display.html?id=79926 > > I believe he modified the select() timeout on line 128 of IBAPI.cxx with > something shorter. >

Thanks cool, I changed the time to 0 and it works fine now. Show quoted text

> Furthermore I keep getting the following error. > > Server Error: ReqID -1; code 509: Exception caught while reading socket - > No such file or directory >

Show quoted text

> That error is generated in MyEventHandler.pm by the 'error()' sub, and it > is called as a result of receiving a message from the server (well, the TWS > endpoint or daemon), so it is something IB is sending to your client. > >

I found this error. It's actually a bug in the IB API but it can be fixed in your module. What happens is the following: * Call eConnect successfully * Other code runs that sets "errno" to a non-zero value, for example due to a non-fatal file-not-found. * After the next successful "select" in handleMessages, it continues to onReceive and then to handleSocketError where it detects a non-zero "errno" and terminates with an error. errno is normally never set to zero and one should only check it's value if a function returns an error. However in the IB API they don't do it that way so it catches an error that really isn't there. The way I got around it is by having the line "errno = 0;" just before the select statement in handleMessages. While I'm at it I have another question. While making your module on OSX it complains about the compiler version and that I should update it. I updated it to the latest but it still complains. I removed this check from Makefile.PL to force building regardless. I don't know what the potential problem was. So do you think I'm still using the wrong compiler or is the check not valid anymore? $ c++ --version i686-apple-darwin11-llvm-g++-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)

Sun Aug 25 11:40:55 2013 INFIDEL [...] cpan.org - Correspondence added

On Sun Aug 25 10:55:47 2013, richie765@gmail.com wrote: Show quoted text

> Thanks cool, I changed the time to 0 and it works fine now.

Great! Though, you may want to check how frequently ping requests are sent now... Show quoted text

> The way I got around it is by having the line "errno = 0;" just before > the > select statement in handleMessages.

Clear as day, thanks for reporting back on this. I assume you mean IBAPI.cxx:processMessages() line 143 here? Show quoted text

> While I'm at it I have another question. While making your module on > OSX it > complains about the compiler version and that I should update it. I > updated > it to the latest but it still complains. I removed this check from > Makefile.PL to force building regardless. > > I don't know what the potential problem was. So do you think I'm still > using the wrong compiler or is the check not valid anymore? > > $ c++ --version > i686-apple-darwin11-llvm-g++-4.2 (GCC) 4.2.1 (Based on Apple Inc. > build 5658) (LLVM build 2336.11.00)

The build-time checks for OS-X were the result of this bug report: https://rt.cpan.org/Ticket/Display.html?id=76932 I spent several hours digging through that, and discovered 2 things: 1. The Apple devs, or whoever builds the Perl binaries for OSX, do not use a system layout that matches that of their end-users, so 'perl -V' and the Config.pm module lie on OS-X, and cause many build headaches, likely with many other complex Perl modules also. 2. Some bug in Apple's XCode 4.1 or 4.2 seems to generate bogus binaries, which caused build failures in many projects. From what I could tell, it seemed to be an issue during their migration to clang/llvm. I'm not an expert in OS-X's toolchain, and don't use it for development myself, so I added those checks after a few days of frustrating Heisenbugs when running identical test cases on different versions of the OS-X toolchain. If it works for you without generating segfaults or other issues, it may be that it's now fixed after all, and my checks should be refined some other way, since your toolchain signature looks the same as the one in that ticket. Don't suppose you could paste your 'perl -V', 'uname -a', and other toolchain and XCode versions, and OS-X version in here (masking any private information)? HTH, and hope everything works well for you now!

Sun Aug 25 12:30:51 2013 INFIDEL [...] cpan.org - Status changed from 'open' to 'patched'

Sun Aug 25 12:30:51 2013 INFIDEL [...] cpan.org - Severity Normal added

Sun Aug 25 12:30:52 2013 INFIDEL [...] cpan.org - Broken in 0.08 added

Sun Aug 25 13:37:05 2013 richie765 [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #88097] Using with a GUI
Date:	Sun, 25 Aug 2013 19:36:49 +0200
To:	bug-Finance-InteractiveBrokers-SWIG [...] rt.cpan.org
From:	Richard <richie765 [...] gmail.com>

<URL: https://rt.cpan.org/Ticket/Display.html?id=88097 > Show quoted text

> >

> > Thanks cool, I changed the time to 0 and it works fine now.

> > Great! Though, you may want to check how frequently ping requests are > sent now... > >

I'll check but I think that hasn't changed. 0 seems to be the best for GUI, I can call handleMessages a couple of times per second when idle and using a timer without blocking the UI at all. So far that works very well. If you are making an update, you could make the timeout configurable. In its current form the module only seems to be usable in a very specific way. Show quoted text

> > Clear as day, thanks for reporting back on this. I assume you mean > IBAPI.cxx:processMessages() line 143 here? > >

Exactly so it should be: errno=0; int ret = select( m_pClient->fd() + 1, &readSet, &writeSet, &errorSet, &tval ); Show quoted text

> > The build-time checks for OS-X were the result of this bug report: > https://rt.cpan.org/Ticket/Display.html?id=76932 > >

Ah ok, I didn't get segfaults so I think that problem is of the past. Perhaps this check isn't so important anymore if you put the requirements in the README. Show quoted text

> > Don't suppose you could paste your 'perl -V', 'uname -a', and other > toolchain and XCode versions, and OS-X version in here (masking any private > information)? > >

I don't think that is a problem. It's at the bottom of this mail. Show quoted text

> HTH, and hope everything works well for you now! >

So far it works great! I'm really happy about this module. Thanks a lot for making it and responding so fast. Everything is up to date to the latest versions. $ perl -V Summary of my perl5 (revision 5 version 12 subversion 4) configuration: Platform: osname=darwin, osvers=12.0, archname=darwin-thread-multi-2level uname='darwin b1026.apple.com 12.0 darwin kernel version 12.0.0: tue may 15 23:31:29 pdt 2012; root:xnu-2050.6.70~1release_x86_64 x86_64 ' config_args='-ds -e -Dprefix=/usr -Dccflags=-g -pipe -Dldflags= -Dman3ext=3pm -Duseithreads -Duseshrplib -Dinc_version_list=none -Dcc=clang' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='clang', ccflags ='-arch i386 -arch x86_64 -g -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protector -I/usr/local/include', optimize='-Os', cppflags='-g -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='clang -mmacosx-version-min=10.8', ldflags ='-arch i386 -arch x86_64 -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /usr/lib libs=-ldbm -ldl -lm -lutil -lc perllibs=-ldl -lm -lutil -lc libc=, so=dylib, useshrplib=true, libperl=libperl.dylib gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-arch i386 -arch x86_64 -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API Locally applied patches: /Library/Perl/Updates/<version> comes before system perl directories installprivlib and installarchlib points to the Updates directory Built under darwin Compiled at Jun 20 2012 13:55:28 %ENV: PERL5LIB="/path/to/lib" @INC: /path/to/lib /Library/Perl/5.12/darwin-thread-multi-2level /Library/Perl/5.12 /Network/Library/Perl/5.12/darwin-thread-multi-2level /Network/Library/Perl/5.12 /Library/Perl/Updates/5.12.4/darwin-thread-multi-2level /Library/Perl/Updates/5.12.4 /System/Library/Perl/5.12/darwin-thread-multi-2level /System/Library/Perl/5.12 /System/Library/Perl/Extras/5.12/darwin-thread-multi-2level /System/Library/Perl/Extras/5.12 . $ uname -a Darwin somehost.local 12.4.0 Darwin Kernel Version 12.4.0: Wed May 1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64 $ gcc --version i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00) XCode: Version 4.6.3 (4H1503) Commandline Tools: Updated to the latest

Mon Aug 26 07:59:15 2013 INFIDEL [...] cpan.org - Correspondence added

On Sun Aug 25 13:37:05 2013, richie765@gmail.com wrote: Show quoted text

> Ah ok, I didn't get segfaults so I think that problem is of the past. > Perhaps this check isn't so important anymore if you put the > requirements > in the README.

In looking at your versions, and redoing the research on it, it looks like it was an issue with fastcall and/or varargs semantics in LLVM build 2335. Many people were complaining loudly about ABI issues in that release, though I still haven't found an authoritative root cause. I changed the signature checker to complain about that, rather than the toolchain string, for v0.09. I also added the 'errno = 0;' reset that you mentioned. If you'd like to try out the build step in 0.09 to see if it complains, feel free (and I'd love to hear back), but for now if you install it, you'd have to re-add your timeout tweak. I'll work on a configurable timeout for a later release when I've got a bit more time. Thanks for the feedback, and happy trading!

Mon Aug 26 07:59:15 2013 INFIDEL [...] cpan.org - Fixed in 0.09 added

Fri Aug 30 12:53:10 2013 INFIDEL [...] cpan.org - Correspondence added

I've uploaded FIB::SWIG 0.10, which adds the method setSelectTimeout(), which should hopefully be backwards compatible, and allow a reset of the select(2) timeout issue for your interactive use case. It has a dependency on FIB::API 0.04, which I also just uploaded. I'd love to know if this works for you.

Fri Aug 30 20:35:12 2013 INFIDEL [...] cpan.org - Fixed in 0.10 added

Sat Sep 07 02:01:43 2013 INFIDEL [...] cpan.org - Status changed from 'patched' to 'resolved'