Bug #118900 for mod_perl: Check.xs Segfault In mod_perl

I am attempting to run an already-bug-free Catalyst application under mod_perl, which should work just fine but instead gives a segfault in the Check.xs file of the B::Hooks::OP::Check distribution. (I have already asked for help from the Check.xs team, and now I'm asking here as well.) The name of the Catalyst application is ShinyCMS, it works just fine using the PSGI stand-alone testing web server as well as fastcgi, and mod_perl is the only option which causes a segfault. I must have mod_perl support for my specific use of ShinyCMS, so I really really need to fix this segfault and get Shiny working under mod_perl. Yes, Shiny is specifically designed to work with mod_perl, as are most Catalyst apps. In fact, my simple "<<< DEBUG >>>" print statements in ShinyCMS.pm show that Shiny is totally loading and returning control back to Apache (or mod_perl?) before the actual segfault occurs. My system info: $ apache2 -v Server version: Apache/2.4.18 (Ubuntu) Server built: 2016-07-14T12:32:26 $ perl -v This is perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-linux-gnu-thread-multi (with 58 registered patches, see perl -V for more detail) $ pm_version.pl mod_perl2 $mod_perl2::VERSION = 2.000009 $ pm_version.pl B::Hooks::OP::Check $B::Hooks::OP::Check::VERSION = 0.19 $ cat /etc/issue Ubuntu 16.04.1 LTS I have spent the last week trying all of the following and more, but the segfault persists: 1. manually-compiled perl v5.22.1 & mod_perl2 v2.000009 w/out threads 2. mod_perl2 back to v2.000005 3. B::Hooks::OP::Check back to v0.17 (maybe earlier) 4. move all possible 'use' statements to be called as early as possible in Shiny_dependencies.pm I also tried Apache v2.2 & mod_perl2 v2.000008 & Perl v5.18.2 in Ubuntu v14.04, it did not segfault but instead it served up totally blank pages with 0 bytes of data. I have not dug deeper because I use Ubuntu v16.04 in production and I figure it is harder to debug w/out an actual segfault to start from. I have set up SSH access on a cloud server so you can see and debug the segfault directly, all 3 of the usernames below are set as sudoers so please wield your root powers with caution: HOSTNAME: cloud-comp0-00.autoparallel.com HOST IP: 64.137.255.53 USERNAME: shay OR fmoyer OR jgao OR modperl_team PASSWORD: check your CPAN e-mail address Pertinent paths: /home/wbraswell/public_html/cloud-comp0-00.autoparallel.com-latest /etc/apache2/sites-available/cloud-comp0-00.autoparallel.com.conf Here is an example gdb showing Check.xs as the likely culprit, starting with `sudo` and then `source` to load Apache env vars: [[[ BEGIN GDB SESSION ]]] zefram@cloud-comp0-00:/$ sudo -i ... root@cloud-comp0-00:~# source /etc/apache2/envvars root@cloud-comp0-00:~# gdb /usr/sbin/apache2 ... Reading symbols from /usr/sbin/apache2...(no debugging symbols found)...done. (gdb) run -k start -X Starting program: /usr/sbin/apache2 -k start -X [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". <<< DEBUG >>>: top of ShinyCMS.pm <<< DEBUG >>>: in ShinyCMS.pm, about to use Moose <<< DEBUG >>>: in ShinyCMS.pm, about to use Catalyst::Runtime <<< DEBUG >>>: in ShinyCMS.pm, about to use Catalyst <<< DEBUG >>>: in ShinyCMS.pm, about to use CatalystX::RoleApplicator <<< DEBUG >>>: in ShinyCMS.pm, about to use Method::Signatures::Simple <<< DEBUG >>>: in ShinyCMS.pm, about to call config() <<< DEBUG >>>: in ShinyCMS.pm, have __PACKAGE__ = ShinyCMS <<< DEBUG >>>: in ShinyCMS.pm, about to call setup()... <<< DEBUG >>>: in ShinyCMS.pm, returned from setup() <<< DEBUG >>>: in ShinyCMS.pm, about to return 1 Program received signal SIGSEGV, Segmentation fault. 0x00007fffebabcb10 in ?? () (gdb) bt #0 0x00007fffebabcb10 in ?? () #1 0x00007ffff3e5302b in Perl_newUNOP (my_perl=my_perl@entry=0x5555577ddde0, type=type@entry=17, flags=<optimized out>, flags@entry=8192, first=0x5555562141b8) at op.c:4811 #2 0x00007ffff3e54a1d in Perl_newCVREF (my_perl=my_perl@entry=0x5555577ddde0, flags=flags@entry=8192, o=<optimized out>) at op.c:9367 #3 0x00007ffff3e8b686 in Perl_yylex (my_perl=my_perl@entry=0x5555577ddde0) at toke.c:6693 #4 0x00007ffff3e97228 in Perl_yyparse (my_perl=my_perl@entry=0x5555577ddde0, gramtype=gramtype@entry=258) at perly.c:322 ... #44 0x000055555558709f in main () (gdb) start The program being debugged has been started already. Start it from the beginning? (y or n) y Temporary breakpoint 1 at 0x5555555867b0 Starting program: /usr/sbin/apache2 -k start -X [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Temporary breakpoint 1, 0x00005555555867b0 in main () (gdb) break perl_parse Function "perl_parse" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 2 (perl_parse) pending. (gdb) continue Continuing. Breakpoint 2, perl_parse (my_perl=0x55555581ae50, xsinit=0x7ffff420a280, argc=2, argv=0x7ffff7f88070, env=0x0) at perl.c:1473 1473 perl.c: No such file or directory. (gdb) watch -l PL_check[17] Hardware watchpoint 3: -location PL_check[17] (gdb) continue Continuing. Hardware watchpoint 3: -location PL_check[17] Old value = (OP *(*)(PerlInterpreter *, OP *)) 0x7ffff3e4d9e0 <Perl_ck_rvconst> New value = (OP *(*)(PerlInterpreter *, OP *)) 0x7fffebabcb10 <check_cb> 0x00007fffebabcd0e in hook_op_check (type=type@entry=OP_RV2CV, cb=cb@entry=0x7fffeb8b6d30 <dd_ck_rv2cv>, user_data=user_data@entry=0x0) at Check.xs:66 66 PL_check[type] = check_cb; (gdb) info threads Id Target Id Frame * 1 Thread 0x7ffff7fe2780 (LWP 6196) "/usr/sbin/apach" 0x00007fffebabcd0e in hook_op_check (type=type@entry=OP_RV2CV, cb=cb@entry=0x7fffeb8b6d30 <dd_ck_rv2cv>, user_data=user_data@entry=0x0) at Check.xs:66 (gdb) info source Current source file is Check.xs Compilation directory is /home/wbraswell/.cpanm/work/1479735542.24787/B-Hooks-OP-Check-0.19 Located in /home/wbraswell/.cpanm/work/1479735542.24787/B-Hooks-OP-Check-0.19/Check.xs Contains 106 lines. Source language is c. Producer is GNU C11 5.4.0 20160609 -mtune=generic -march=x86-64 -g -O2 -fwrapv -fno-strict-aliasing -fPIC -fstack-protector-strong. Compiled with DWARF 2 debugging format. Does not include preprocessor macro info. (gdb) continue Continuing. <<< DEBUG >>>: top of ShinyCMS.pm <<< DEBUG >>>: in ShinyCMS.pm, about to use Moose <<< DEBUG >>>: in ShinyCMS.pm, about to use Catalyst::Runtime <<< DEBUG >>>: in ShinyCMS.pm, about to use Catalyst <<< DEBUG >>>: in ShinyCMS.pm, about to use CatalystX::RoleApplicator <<< DEBUG >>>: in ShinyCMS.pm, about to use Method::Signatures::Simple <<< DEBUG >>>: in ShinyCMS.pm, about to call config() <<< DEBUG >>>: in ShinyCMS.pm, have __PACKAGE__ = ShinyCMS <<< DEBUG >>>: in ShinyCMS.pm, about to call setup()... <<< DEBUG >>>: in ShinyCMS.pm, returned from setup() <<< DEBUG >>>: in ShinyCMS.pm, about to return 1 ... Program received signal SIGSEGV, Segmentation fault. 0x00007fffebabcb10 in ?? () (gdb) bt #0 0x00007fffebabcb10 in ?? () #1 0x00007ffff3e5302b in Perl_newUNOP (my_perl=my_perl@entry=0x555559dd8ed0, type=type@entry=17, flags=<optimized out>, flags@entry=8192, first=0x55555824eb98) at op.c:4811 #2 0x00007ffff3e54a1d in Perl_newCVREF (my_perl=my_perl@entry=0x555559dd8ed0, flags=flags@entry=8192, o=<optimized out>) at op.c:9367 ... #44 0x000055555558709f in main () (gdb) print PL_check[17] $1 = (Perl_check_t) 0x7fffebabcb10 (gdb) quit [[[ END GDB SESSION ]]] I have attached the unabbreviated gdb session and the system info in text files, for your reference. I am anxious (near desperate) to get this bug fixed so that I can move forward with my work. I will eagerly assist in any way I can, please just let me know what I should do and I will do it! Thanks so much for your help. Perling, ~ Will