Skip Menu |

This queue is for tickets about the B-Hooks-OP-Check CPAN distribution.

Report information
The Basics
Id: 118899
Status: open
Priority: 0/
Queue: B-Hooks-OP-Check
People
Owner: ZEFRAM [...] cpan.org
Requestors: WBRASWELL [...] cpan.org
Cc:
AdminCc:
Bug Information
Severity: Critical
Broken in:
  • 0.17
  • 0.18
  • 0.19
Fixed in: (no value)

History Show all quoted text
  Tue Nov 22 04:47:11 2016 WBRASWELL [...] cpan.org - Ticket created
Subject: Check.xs Segfault In mod_perl
Hello Everyone, I am attempting to run an already-bug-free Catalyst application under mod_perl, which should work just fine but instead gives a segfault in the Check.xs file. The name of the Catalyst application is ShinyCMS, it works just fine using the PSGI stand-alone testing web server as well as fastcgi, and mod_perl is the only option which causes a segfault. Unfortunately, I must have mod_perl support for my specific use of ShinyCMS, so I really really need to fix this segfault and get Shiny working under mod_perl. Yes, Shiny is specifically designed to work with mod_perl, as are most Catalyst apps. In fact, my simple "<<< DEBUG >>>" print statements in ShinyCMS.pm show that Shiny is totally loading and returning control back to Apache (or mod_perl?) before the actual segfault occurs. My system info: $ apache2 -v Server version: Apache/2.4.18 (Ubuntu) Server built: 2016-07-14T12:32:26 $ perl -v This is perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-linux-gnu-thread-multi (with 58 registered patches, see perl -V for more detail) $ pm_version.pl mod_perl2 $mod_perl2::VERSION = 2.000009 $ pm_version.pl B::Hooks::OP::Check $B::Hooks::OP::Check::VERSION = 0.19 $ cat /etc/issue Ubuntu 16.04.1 LTS I have spent the last week trying all of the following and more, but the segfault persists: 1. manually-compiled perl v5.22.1 & mod_perl2 v2.000009 w/out threads 2. mod_perl2 back to v2.000005 3. B::Hooks::OP::Check back to v0.17 (maybe earlier) 4. move all possible 'use' statements to be called as early as possible in Shiny_dependencies.pm I also tried Apache v2.2 & mod_perl2 v2.000008 & Perl v5.18.2 in Ubuntu v14.04, it did not segfault but instead it served up totally blank pages with 0 bytes of data. I have not dug deeper because I use Ubuntu v16.04 in production and I figure it is harder to debug w/out an actual segfault to start from. I have set up SSH access on a cloud server so you can see and debug the segfault directly, all 3 of the usernames below are set as sudoers so please wield your root powers with caution: HOSTNAME: cloud-comp0-00.autoparallel.com HOST IP: 64.137.255.53 USERNAME: zefram OR flora OR bhooks_team PASSWORD: check your CPAN e-mail address Pertinent paths: /home/wbraswell/public_html/cloud-comp0-00.autoparallel.com-latest /etc/apache2/sites-available/cloud-comp0-00.autoparallel.com.conf Here is an example gdb showing Check.xs as the likely culprit, starting with `sudo` and then `source` to load Apache env vars: [[[ BEGIN GDB SESSION ]]] zefram@cloud-comp0-00:/$ sudo -i ... root@cloud-comp0-00:~# source /etc/apache2/envvars root@cloud-comp0-00:~# gdb /usr/sbin/apache2 ... Reading symbols from /usr/sbin/apache2...(no debugging symbols found)...done. (gdb) run -k start -X Starting program: /usr/sbin/apache2 -k start -X [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". <<< DEBUG >>>: top of ShinyCMS.pm <<< DEBUG >>>: in ShinyCMS.pm, about to use Moose <<< DEBUG >>>: in ShinyCMS.pm, about to use Catalyst::Runtime <<< DEBUG >>>: in ShinyCMS.pm, about to use Catalyst <<< DEBUG >>>: in ShinyCMS.pm, about to use CatalystX::RoleApplicator <<< DEBUG >>>: in ShinyCMS.pm, about to use Method::Signatures::Simple <<< DEBUG >>>: in ShinyCMS.pm, about to call config() <<< DEBUG >>>: in ShinyCMS.pm, have __PACKAGE__ = ShinyCMS <<< DEBUG >>>: in ShinyCMS.pm, about to call setup()... <<< DEBUG >>>: in ShinyCMS.pm, returned from setup() <<< DEBUG >>>: in ShinyCMS.pm, about to return 1 Program received signal SIGSEGV, Segmentation fault. 0x00007fffebabcb10 in ?? () (gdb) bt #0 0x00007fffebabcb10 in ?? () #1 0x00007ffff3e5302b in Perl_newUNOP (my_perl=my_perl@entry=0x5555577ddde0, type=type@entry=17, flags=<optimized out>, flags@entry=8192, first=0x5555562141b8) at op.c:4811 #2 0x00007ffff3e54a1d in Perl_newCVREF (my_perl=my_perl@entry=0x5555577ddde0, flags=flags@entry=8192, o=<optimized out>) at op.c:9367 #3 0x00007ffff3e8b686 in Perl_yylex (my_perl=my_perl@entry=0x5555577ddde0) at toke.c:6693 #4 0x00007ffff3e97228 in Perl_yyparse (my_perl=my_perl@entry=0x5555577ddde0, gramtype=gramtype@entry=258) at perly.c:322 ... #44 0x000055555558709f in main () (gdb) start The program being debugged has been started already. Start it from the beginning? (y or n) y Temporary breakpoint 1 at 0x5555555867b0 Starting program: /usr/sbin/apache2 -k start -X [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Temporary breakpoint 1, 0x00005555555867b0 in main () (gdb) break perl_parse Function "perl_parse" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 2 (perl_parse) pending. (gdb) continue Continuing. Breakpoint 2, perl_parse (my_perl=0x55555581ae50, xsinit=0x7ffff420a280, argc=2, argv=0x7ffff7f88070, env=0x0) at perl.c:1473 1473 perl.c: No such file or directory. (gdb) watch -l PL_check[17] Hardware watchpoint 3: -location PL_check[17] (gdb) continue Continuing. Hardware watchpoint 3: -location PL_check[17] Old value = (OP *(*)(PerlInterpreter *, OP *)) 0x7ffff3e4d9e0 <Perl_ck_rvconst> New value = (OP *(*)(PerlInterpreter *, OP *)) 0x7fffebabcb10 <check_cb> 0x00007fffebabcd0e in hook_op_check (type=type@entry=OP_RV2CV, cb=cb@entry=0x7fffeb8b6d30 <dd_ck_rv2cv>, user_data=user_data@entry=0x0) at Check.xs:66 66 PL_check[type] = check_cb; (gdb) info threads Id Target Id Frame * 1 Thread 0x7ffff7fe2780 (LWP 6196) "/usr/sbin/apach" 0x00007fffebabcd0e in hook_op_check (type=type@entry=OP_RV2CV, cb=cb@entry=0x7fffeb8b6d30 <dd_ck_rv2cv>, user_data=user_data@entry=0x0) at Check.xs:66 (gdb) info source Current source file is Check.xs Compilation directory is /home/wbraswell/.cpanm/work/1479735542.24787/B-Hooks-OP-Check-0.19 Located in /home/wbraswell/.cpanm/work/1479735542.24787/B-Hooks-OP-Check-0.19/Check.xs Contains 106 lines. Source language is c. Producer is GNU C11 5.4.0 20160609 -mtune=generic -march=x86-64 -g -O2 -fwrapv -fno-strict-aliasing -fPIC -fstack-protector-strong. Compiled with DWARF 2 debugging format. Does not include preprocessor macro info. (gdb) continue Continuing. <<< DEBUG >>>: top of ShinyCMS.pm <<< DEBUG >>>: in ShinyCMS.pm, about to use Moose <<< DEBUG >>>: in ShinyCMS.pm, about to use Catalyst::Runtime <<< DEBUG >>>: in ShinyCMS.pm, about to use Catalyst <<< DEBUG >>>: in ShinyCMS.pm, about to use CatalystX::RoleApplicator <<< DEBUG >>>: in ShinyCMS.pm, about to use Method::Signatures::Simple <<< DEBUG >>>: in ShinyCMS.pm, about to call config() <<< DEBUG >>>: in ShinyCMS.pm, have __PACKAGE__ = ShinyCMS <<< DEBUG >>>: in ShinyCMS.pm, about to call setup()... <<< DEBUG >>>: in ShinyCMS.pm, returned from setup() <<< DEBUG >>>: in ShinyCMS.pm, about to return 1 ... Program received signal SIGSEGV, Segmentation fault. 0x00007fffebabcb10 in ?? () (gdb) bt #0 0x00007fffebabcb10 in ?? () #1 0x00007ffff3e5302b in Perl_newUNOP (my_perl=my_perl@entry=0x555559dd8ed0, type=type@entry=17, flags=<optimized out>, flags@entry=8192, first=0x55555824eb98) at op.c:4811 #2 0x00007ffff3e54a1d in Perl_newCVREF (my_perl=my_perl@entry=0x555559dd8ed0, flags=flags@entry=8192, o=<optimized out>) at op.c:9367 ... #44 0x000055555558709f in main () (gdb) print PL_check[17] $1 = (Perl_check_t) 0x7fffebabcb10 (gdb) quit [[[ END GDB SESSION ]]] I have attached the unabbreviated gdb session and the system info in text files, for your reference. I am anxious (near desperate) to get this bug fixed so that I can move forward with my work. I will eagerly assist in any way I can, please just let me know what I should do and I will do it! Thanks so much for your help. Perling, ~ Will
Subject: segfault_gdb.txt

Message body is not shown because it is too large.

Subject: segfault_versions.txt
wbraswell@cloud-comp0-00:~$ apache2 -v Server version: Apache/2.4.18 (Ubuntu) Server built: 2016-07-14T12:32:26 wbraswell@cloud-comp0-00:~$ perl -v This is perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-linux-gnu-thread-multi (with 58 registered patches, see perl -V for more detail) wbraswell@cloud-comp0-00:~$ pm_version.pl mod_perl2 $mod_perl2::VERSION = 2.000009 wbraswell@cloud-comp0-00:~$ pm_version.pl B::Hooks::OP::Check $B::Hooks::OP::Check::VERSION = 0.19 wbraswell@cloud-comp0-00:~$ cat /etc/issue Ubuntu 16.04.1 LTS
  Tue Nov 22 09:18:20 2016 WBRASWELL [...] cpan.org - Correspondence added
I forgot to mention I also have already tried mod_perl v2.0.10 on my personal (non-cloud) testing system and I was still getting the segfault. I have now updated mod_perl on the cloud server from v2.0.9 to v2.0.10, and again we are still getting the same segfault as before. So now you can log into the cloud server and use gdb to see the segfault in action, you should have a private e-mail with the SSH login credentials, if not please let me know. Thanks, ~ Will
  Tue Dec 13 06:45:03 2016 HJANSEN [...] cpan.org - Correspondence added
We´re using a custom multithreaded C++ application in which the original developers embedded a perl interpreter as a means to extend the functionality implemented in C++. Most of the time executing the perl code works just fine. But sometimes we see a segfault and in these cases when walking from the lowest stack entry upwards it seems that "check_cb" in Check.xs from B::Hooks::OP::Check is the first common element. Some abbreviated stacks to illustrate this: #0 S_mg_findext_flags (my_perl=0x2b1e8e0fd480, flags=0, vtbl=0x0, type=type@entry=126, sv=sv@entry=0x6d702e65636e6174) at mg.c:400 #1 Perl_mg_find (my_perl=0x2b1e8e0fd480, sv=sv@entry=0x6d702e65636e6174, type=type@entry=126) at mg.c:421 #2 0x00002b1ee98cdc92 in get_mg_ptr (sv=0x6d702e65636e6174) at Check.xs:20 #3 0x00002b1ee98cdd77 in check_cb (my_perl=<optimized out>, op=<optimized out>) at Check.xs:46 #0 0x00002b1e82642d59 in S_av_top_index (av=av@entry=0x2b1e8e2411b0, my_perl=0x2b1e8dc72aa0) at inline.h:23 #1 Perl_av_len (my_perl=0x2b1e8dc72aa0, av=av@entry=0x2b1e8e2411b0) at av.c:764 #2 0x00002b1ee98cdd46 in check_cb (my_perl=<optimized out>, op=<optimized out>) at Check.xs:37 #0 0x00002b1e826423a3 in Perl_av_fetch (my_perl=0x2b1eadcccf30, av=av@entry=0x2b1eae2410f0, key=key@entry=0, lval=lval@entry=0) at av.c:253 #1 0x00002b1ee98cdd62 in check_cb (my_perl=<optimized out>, op=<optimized out>) at Check.xs:40 We´re using B::Hooks::OP::Check 0.19. Perl is "perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux-thread-multi" as provided by "SUSE Linux Enterprise Server 12". So it might be that we see the same error as reported here. Unfortunately my C++ and XS knowledge is rather limited. Please let me know if can do anything else to help finding out what is going on and if our issue is really related to this bug report.
  Tue Dec 13 06:45:04 2016 The RT System itself - Status changed from 'new' to 'open'
  Mon Dec 19 06:59:50 2016 HJANSEN [...] cpan.org - Correspondence added
So in our case we found that B::Hooks::Check::OP was included only fairly recently as a dependency of TryCatch. Commenting all "try { ... } catch { ... }" statements did not help unless we also commented all "use TryCatch;" statements. But once we´d done that, the app stopped dumping SEGV cores!! I´m not sure if the error lies with how TryCatch uses B::Hooks::Check::OP or in B::Hooks::Check::OP itself; since I haven´t seen SEGVs in a Mojolicious app which makes use of TryCatch, executing the perl code through an interpreter embedded in a (multithreaded?) C/C++ software might be the distinguishing factor.
  Tue Dec 27 16:11:08 2016 WBRASWELL [...] cpan.org - Correspondence added
Hi Heiko, I appreciate your response & possible solution! Yes, your problem sounds very very similar. For the time being, I have stopped trying to get mod_perl to work; however, this bug remains active and I may have to revisit it at some point in the future if I ever need to get my application working in mod_perl for some reason. I hope Zefram and/or the other admins will look into this for the sake of anyone else who may suffer from this bug down the road. Thanks everyone! Perling, ~ Will the Chill On Mon Dec 19 06:59:50 2016, HJANSEN wrote: Show quoted text
> So in our case we found that B::Hooks::Check::OP was included only > fairly recently as a dependency of TryCatch. > > Commenting all "try { ... } catch { ... }" statements did not help > unless we also commented all "use TryCatch;" statements. But once we´d > done that, the app stopped dumping SEGV cores!! > > I´m not sure if the error lies with how TryCatch uses > B::Hooks::Check::OP or in B::Hooks::Check::OP itself; since I haven´t > seen SEGVs in a Mojolicious app which makes use of TryCatch, executing > the perl code through an interpreter embedded in a (multithreaded?) > C/C++ software might be the distinguishing factor.
  Mon Jan 02 06:40:32 2017 HJANSEN [...] cpan.org - Correspondence added
Am Di 27. Dez 2016, 16:11:08, WBRASWELL schrieb: Show quoted text
> For the time being, I have stopped trying to get mod_perl to work; > however, this bug remains active and I may have to revisit it at some > point in the future if I ever need to get my application working in > mod_perl for some reason.
I´m considering switching to https://metacpan.org/pod/Syntax::Feature::Try as its feature set is quite similar to TryCatch while it is implemented using the perl keyword API instead of Devel::Declare. Best - Heiko
  Mon Jan 02 09:17:13 2017 wbraswell_cpan [...] nym.hush.com - Correspondence added
Subject: Re: [rt.cpan.org #118899] Check.xs Segfault In mod_perl
Date: Mon, 02 Jan 2017 08:17:02 -0600
To: bug-b-hooks-op-check [...] rt.cpan.org
From: wbraswell_cpan [...] nym.hush.com
Good info, thanks! ~ Will On 1/2/2017 at 5:40 AM, "Heiko Jansen via RT" wrote: Am Di 27. Dez 2016, 16:11:08, WBRASWELL schrieb: Show quoted text
> For the time being, I have stopped trying to get mod_perl to work; > however, this bug remains active and I may have to revisit it at
some Show quoted text
> point in the future if I ever need to get my application working in > mod_perl for some reason.
I´m considering switching to https://metacpan.org/pod/Syntax::Feature::Try as its feature set is quite similar to TryCatch while it is implemented using the perl keyword API instead of Devel::Declare. Best - Heiko
  Mon Jan 02 13:59:08 2017 ether [...] cpan.org - Correspondence added
On 2017-01-02 03:40:32, HJANSEN wrote: Show quoted text
> Am Di 27. Dez 2016, 16:11:08, WBRASWELL schrieb: >
> > For the time being, I have stopped trying to get mod_perl to work; > > however, this bug remains active and I may have to revisit it at some > > point in the future if I ever need to get my application working in > > mod_perl for some reason.
> > I´m considering switching to > https://metacpan.org/pod/Syntax::Feature::Try as its feature set is > quite similar to TryCatch while it is implemented using the perl > keyword API instead of Devel::Declare.
Have you considered Try::Tiny? (I'm not the author.) Nevertheless, it's certainly a good idea to move off of anything implemented using Devel::Declare.
  Mon Jul 31 06:30:04 2017 ZEFRAM [...] cpan.org - Correspondence added
B::Hooks::OP::Check has never been thread-safe, and you're running into the problem that arises from that. This might change in a future version. -zefram