Bug #118657 for Prima: Prima won't install on plain windows: img\codec_X11.c:7:23: fatal error: X11/Xlib.h: No such file or directory

Sun Nov 06 20:00:49 2016 MITHALDU [...] cpan.org - Ticket created

Subject:

Prima won't install on plain windows: img\codec_X11.c:7:23: fatal error: X11/Xlib.h: No such file or directory

I installed the codec pack mentioned in the readme (should be mentioned in the main module, and maybe even get a prompt when mfpl is interactive). Yet still i get this error: gcc -c -Iinclude -Iinclude/generic -fopenmp -s -O2 -DWIN32 -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fno-strict-aliasing -mms-bitfields -s -O2 -DVERSION=\"1.49\" -DXS_VERSION=\"1.49\" "-IC:\strawberry\perl\lib\CORE" img\codec_X11.c -o img\codec_X11.o img\codec_X11.c:7:23: fatal error: X11/Xlib.h: No such file or directory compilation terminated. dmake.EXE: Error code 129, while making 'img\codec_X11.o' Full details: https://gist.github.com/wchristian/6f6fa73a46950b988937c8d20476faec

Mon Nov 07 01:34:25 2016 KARASIK [...] cpan.org - Correspondence added

Hi Christian, It looks that you have already a x11 installed, which gets in conflict with Prima: Show quoted text

> Checking for x11 using pkg-config...1.6.3

I believe it might be misconfigured; you can try to run "pkg-config --cflags-only-I x11" on your system to see if it has X11.h there. Dmitry

Mon Nov 07 01:34:25 2016 The RT System itself - Status changed from 'new' to 'open'

Mon Nov 07 01:48:17 2016 KARASIK [...] cpan.org - Correspondence added

Oh, this is not plain windows, this is strawberry -- it should have all graphic libraries installed by default (you only need the codec pack for activestate perl). Is it by chance a pre-5.20 strawberry? Prima still should be working on them with no problem (it had pkgconfig problem fixed in 5.20), but could you possible give me the strawberry version so I could try to reproduce? /dk

Mon Nov 07 02:01:05 2016 KARASIK [...] cpan.org - Correspondence added

Hmm no, just tried on sb-5.18.1, all is good. Looks like a pkgconfig issue.

Mon Nov 07 09:43:57 2016 MITHALDU [...] cpan.org - Correspondence added

It is Strawberry 18.4. Here's an archive of it: https://dl.dropboxusercontent.com/u/10190786/strawberry.rar Also, i meant plain because it's not cygwin, and at the current time strawberry *is* the de factor default perl on windows.

Mon Nov 07 11:17:07 2016 KARASIK [...] cpan.org - Correspondence added

Just tried, all good as well. Also, the archive doesn't have any pkg-config.bat which was used by Makefile.PL as seen in the log file you've sent -- are you sure it was not dragged in by your non-strawberry setup?

Mon Nov 07 12:08:03 2016 KARASIK [...] cpan.org - Correspondence added

(btw if you don't want to waste time on debugging you can just 'rm img/codec_X11.c' and re-run Makefile.PL etc)

Mon Nov 07 12:58:21 2016 MITHALDU [...] cpan.org - Correspondence added

I'm fine with debugging. As for pkg-config: I have set up cygwin to be visible from cmd shell by default and it looks like it's finding that one: C:\Users\Mithaldu>which pkg-config /usr/bin/pkg-config Maybe your mfpl needs to recognize that and ignore pkg-config if it's a cygwin one when being run by a normal windows perl?

Mon Nov 07 13:14:14 2016 KARASIK [...] cpan.org - Correspondence added

Show quoted text

> Maybe your mfpl needs to recognize that and ignore pkg-config if it's > a cygwin one when being run by a normal windows perl?

It could be a valid thing to do, but the problem is that modern strawberries (>=5.20) also have pkg-config included :) I don't think that differing between pkg-config.bat and pkg-config.exe is a good idea. I also think that generally mixing up cygwin and native win32 tools is a dangerous endeavour - f.ex. consider how find.exe is different there -- if you mix one with another, then all bets are off... I can add a non-default option to mfpl to ignore pkg-config, if that helps you though, something like "perl Makefile.PL USE_PKGCONFIG=0" .

Mon Nov 07 14:00:03 2016 MITHALDU [...] cpan.org - Correspondence added

Show quoted text

> pkg-config.bat

It is not pkg-config.bat. It is a cygwin executable built to run primarily within the cygwin environment. Show quoted text

> I also think that generally mixing up cygwin and native win32 tools is > a dangerous endeavour - f.ex. consider how find.exe is different there > -- if you mix one with another, then all bets are off...

Doing this is a very normal and standard way of installing cygwin and recommended in many cygwin install guides. You're almost guaranteed to run into this with a person who has perl and cygwin on their windows. (The find thing is also a red herring since the cygwin find does not override the windows find, and inside the cygwin shell environment the cygwin path comes first, so windows find isn't mixed into the process.) Show quoted text

> I can add a non-default option to mfpl to ignore pkg-config, if that > helps you though, something like "perl Makefile.PL USE_PKGCONFIG=0" .

That would help me, but i'm not interested in that. I would prefer to see a general solution for all other users as well. Instead of dumping it on the user this would be fine and unproblematic: $use_pkgconfig = 0 # cygwin pkg-config on win32 perl if $^O eq 'MSWin32' and `which pkg-config 2>nul` =~ m@^/@;

Mon Nov 07 14:15:12 2016 KARASIK [...] cpan.org - Correspondence added

On Mon Nov 07 14:00:03 2016, MITHALDU wrote: Show quoted text

> > pkg-config.bat

> > It is not pkg-config.bat. It is a cygwin executable built to run > primarily within the cygwin environment.

Yes, I was referring to strawberry's one - it's .bat (vs cygwin's .exe) Show quoted text

> You're almost guaranteed to run into this with a person who has perl and cygwin on their windows.

Well I don't agree to that. Cygwin by default doesn't install itself into PATH, while strawberry does. From Makefile.pl's point of view, all tools it runs should be properly configured by perl, -- it calls gcc and ld, so if one mixes them, that wouldn't work either. Show quoted text

> That would help me, but i'm not interested in that. I would prefer to > see a general solution for all other users as well. Instead of dumping > it on the user this would be fine and unproblematic: > > $use_pkgconfig = 0 # cygwin pkg-config on win32 perl > if $^O eq 'MSWin32' and `which pkg-config 2>nul` =~ m@^/@;

I would say the general solution is to upgrade strawberry to 5.20 as it has this problem fixed. Also, this solution won't work on 5.18 strawberry because "which" is not found in it, and it will make things worse on 5.20 mixed with cygwin because it won't let Prima use the legitimate pkg-config from strawberry. Generally, I'm not against fixing this problem. It's just I don't see a good reliable way to detect cygwin vs non-cygwin. If there was an unambiguous way to tell one from another, f.ex. 'test [ pkg-config -v ] == "cygwin"' I would just apply the fix and be done with it.

Mon Nov 07 14:37:39 2016 MITHALDU [...] cpan.org - Correspondence added

`which`not existing is no problem, since it comes default with cygwin. If there's a cygwin pkg-config around, then there's a cygwin which, and if there's no cygwin pkg-config, then the which isn't there either. The 2>nul takes care of the error message about "executable not found". That said, i did forget that which will always give the path in the cygwin notation, i.e.: C:\Users\Mithaldu>which pkg-config /usr/bin/pkg-config vs C:\Users\Mithaldu>which pkg-config /cygdrive/c/strawberry/perl/bin/pkg-config I think it would be fine to modify the code to this: $use_pkgconfig = 0 # cygwin pkg-config on win32 perl if $^O eq 'MSWin32' and `which pkg-config 2>nul` =~ m@^/usr/bin/pkg-config@; On the other hand, strawberry's pkg-config is very explicit about what it is: C:\Users\Mithaldu>pkg-config --help Name: PkgConfig - Pure-Perl Core-Only replacement for pkg-config Versus cygwin's: C:\Users\Mithaldu>pkg-config --help Usage: pkg-config [OPTION...] Help Options: You could also match on the --help header under Strawberry. Strawberry pkg-config also has this option, which the cygwin one lacks: C:\Users\Mithaldu>pkg-config --real-version ppkg-config - cruftless pkg-config

Mon Nov 07 15:33:07 2016 KARASIK [...] cpan.org - Correspondence added

All right, if you insist :) I totally don't buy it that there are more than one person in the world running 5.18 AND cygwin, but okay -- I have committed the patch, please check out the latest github version if it works for you: https://github.com/dk/Prima /dk

Mon Nov 07 16:18:09 2016 MITHALDU [...] cpan.org - Correspondence added

On Mon Nov 07 21:33:07 2016, KARASIK wrote: Show quoted text

> All right, if you insist :)

Thanks for indulging me. :) Show quoted text

> I totally don't buy it that there are more than one person in the > world running 5.18 AND cygwin, but okay -- I have committed the patch, > please check out the latest github version if it works for you:

Well, running Cygwin like that is much-recommended, and people being lazy is natural human trait. ;) Show quoted text

> https://github.com/dk/Prima

Hrm, this compiled fine, but now the tests segfault *immediately*. In this gist are four logs where i progressively remove more things, up to and including cygwin, but still get crashes during `dmake test`. https://gist.github.com/wchristian/755daa0fba1cf5611255f5d5285b6209

Mon Nov 07 16:39:22 2016 KARASIK [...] cpan.org - Correspondence added

Show quoted text

> Hrm, this compiled fine, but now the tests segfault *immediately*. In > this gist are four logs where i progressively remove more things, up > to and including cygwin, but still get crashes during `dmake test`. > > https://gist.github.com/wchristian/755daa0fba1cf5611255f5d5285b6209

This is crap. But, I really can't reproduce it with 5.18-4, everything works just fine for me, all tests passed, no problem at all. Could you possibly, just for the sake of a clear experiment, remove cygwin from PATH completely and re-build? If some wrong tools were used, everything could be possible.

Mon Nov 07 16:43:01 2016 MITHALDU [...] cpan.org - Correspondence added

Show quoted text

> Could you possibly, just for the sake of a clear experiment, remove > cygwin from PATH completely and re-build?

That is exactly what i did in the third log. :) It also happens under 5.24 with cygwin removed: https://gist.github.com/wchristian/8e6f97b290de15f498fbfbb084e62091

Mon Nov 07 17:05:04 2016 KARASIK [...] cpan.org - Correspondence added

Well... I've never seen anything like that before. This must be something rather low-level. Does 'perl -Mblib -MPrima -e 1' coredump as well? If yes, then I'm throwing a towel - the last ever thing to check if "ldd blib\arch\auto\Prima\Prima.xs.dll" makes any sense. Other than this, I'm lost, am afraid...

Mon Nov 07 17:14:24 2016 MITHALDU [...] cpan.org - Correspondence added

It does indeed. I've not been able to find useful stacktraces in the event viewer or with the visual studio debugger. I do remember Prima worked on my system once, so i will try and bisect tomorrow.

Tue Nov 08 01:44:32 2016 KARASIK [...] cpan.org - Correspondence added

I would suspect the linked libraries - removing codecs (the same way you did with X11.c) and using WITH_OPENMP=0 for minimal setup might hint something

Thu Nov 10 13:53:26 2016 MITHALDU [...] cpan.org - Correspondence added

Good news, after noticing that the CPAN 1.49 doesn't immediately fall over, i bisected things down: https://rt.cpan.org/Ticket/Display.html?id=118657 GOOD: 1c7094ea8d7698245b0045ced2d06ad96136ee93 BAD: 3038c3756140448daa432d52add6e65ce647fa87

Thu Nov 10 14:54:44 2016 KARASIK [...] cpan.org - Correspondence added

Thank you! However, it is still mysterious - it looks like functions returned by LoadLibrary are bad. Those are new Win8 functions that might or might not be available on the system ... What is the OS version you are using? Do you think it would be possible to add some debug info to find out where the crash occurs, and why?

Thu Nov 10 15:01:49 2016 MITHALDU [...] cpan.org - Correspondence added

I'm on windows 10 64 bit pro. I'll do some print debugging to see if i can pinpoint it.

Thu Nov 10 15:22:07 2016 MITHALDU [...] cpan.org - Correspondence added

Does this help you any? https://github.com/wchristian/Prima/commit/59dc79fe63aa6bba5ba958a94c4ca6312be1b574 I think it might be triggering some event after leaving the dpi change function, and then crashes in that event, but i don't know how to figure out where in the code that might be.

Thu Nov 10 15:37:07 2016 KARASIK [...] cpan.org - Correspondence added

Thank you! It helps, somewhat, it seems that the stack is being royally fucked up. Is your perl also 64 bit? Just 5 minutes ago I also tried on Windows 10 64-bit (non-Pro though) and all went just fine.

Thu Nov 10 15:40:16 2016 MITHALDU [...] cpan.org - Correspondence added

It's apparently 32 bit with 64bit ints? This is perl 5, version 24, subversion 0 (v5.24.0) built for MSWin32-x86-multi-thread-64int Let me know if i can be helpful in any other way. :)

Thu Nov 10 15:57:23 2016 KARASIK [...] cpan.org - Correspondence added

Thanks for the research! I can reproduce the problem with 32-bit perl. It's crazy :) I'll keep you in touch.

Thu Nov 10 16:24:48 2016 KARASIK [...] cpan.org - Correspondence added

I think I found it. Could you possibly test the latest github snapshot? Thank you!

Thu Nov 10 17:06:38 2016 MITHALDU [...] cpan.org - Correspondence added

Yep, that did it! Mind explaining what that change does? Just a little curious. :) Also, this test fail is left: t/Object/Fonts.t ........ 796/1274 # Failed test 'text wrap XVMSymbol' # at t/Object/Fonts.t line 75. # '3' # > # '4' # Looks like you failed 1 test of 1274. t/Object/Fonts.t ........ Dubious, test returned 1 (wstat 256, 0x100)

Fri Nov 11 04:04:00 2016 KARASIK [...] cpan.org - Correspondence added

On Thu Nov 10 17:06:38 2016, MITHALDU wrote: Show quoted text

> Yep, that did it! Mind explaining what that change does? Just a little > curious. :)

The change applies a very special hint to compiler to generate the C call conforming to Windows API convention (__stdcall), because apparently by default C compilers use __cdecl. You can read more on https://en.wikipedia.org/wiki/X86_calling_conventions if you're curious, but TL;DR is basically that if you call a function as one while it is another, the stack will be corrupted, which we just observed. Strangely there were no problem on 64-bit perl calling native 64-bit DLL, but with 32-bit perl using WOW64 layer it just happened. Show quoted text

> Also, this test fail is left: > > t/Object/Fonts.t ........ 796/1274 > # Failed test 'text wrap XVMSymbol' > # at t/Object/Fonts.t line 75. > # '3' > # > > # '4' > # Looks like you failed 1 test of 1274. > t/Object/Fonts.t ........ Dubious, test returned 1 (wstat 256, 0x100)

This is very mild, but for pedantry sake I'd like to ask you to upload the offending xvmsymbol.ttf somewhere so I could check out what went wrong.

Fri Nov 11 04:15:34 2016 MITHALDU [...] cpan.org - Correspondence added

On Fri Nov 11 10:04:00 2016, KARASIK wrote: Show quoted text

> [...]

Thanks for the explanation and the link for further reading. It makes a lot more sense now. :) Show quoted text

> This is very mild, but for pedantry sake I'd like to ask you to upload > the offending xvmsymbol.ttf somewhere so I could check out what went > wrong.

Ah, that's a font containing icons for a World of Tanks mod. :D https://dl.dropboxusercontent.com/u/10190786/xvm.zip

Sat Nov 12 05:02:12 2016 KARASIK [...] cpan.org - Correspondence added

Yes, that was an interesting font :) I made the test to assume less, so it shouldn't fail now. Luckily, that was never a bug, just the test expecting too much.

Sat Nov 12 05:13:41 2016 MITHALDU [...] cpan.org - Correspondence added

Ha, cheers! Result: PASS

Sat Nov 12 05:37:53 2016 KARASIK [...] cpan.org - Correspondence added

On Sat Nov 12 05:13:41 2016, MITHALDU wrote: Show quoted text

> Ha, cheers! > > Result: PASS

Cool! I guess we close the issue then $)

Sat Nov 12 05:37:53 2016 KARASIK [...] cpan.org - Status changed from 'open' to 'resolved'