Hi,
On Monday, March 19, 2012 05:31:42 AM you wrote:
Show quoted text
Thanks!
Value your feedback.
Show quoted text> > It appears that in the tst situation described below, the
> > CLONE(...) and END(...) functions are called multiple times
> > during web server startup.
>
> Multiple calls of CLONE() of IMHO OK (it is called for every new thread) and
> should not cause a crash, however multiple calls to END() should not happen
> - and as far as I have tested (I really spend some time on this) the END()
> is called only once when perl interpret is finished (with one exception
> related to perl's fork() emulation on MS Windows platform - but we have a
> workaround for this).
Yes, I saw that.
But, according to
http://perldoc.perl.org/threads.html
"These END blocks will then be executed when the thread's interpreter is
destroyed (i.e., either during a ->join() call, or at program termination)."
I havent been able to generate double END calls in my simple sample code, but
I have proved that Apach2 mod_perl certainly does call END multiple times
Show quoted text>
> Anyhow, I do not a reason why END(...) should be called at webserver startup
> (I expect it to be called at shutdown)
>
> See
http://perldoc.perl.org/perlmod.html - quote:
>
> "An END code block is executed as late as possible, that is, after perl has
> finished running the program and just before the interpreter is being
> exited, even if it is exiting as a result of a die() function."
>
> > ...
> >
> > Further,
> > openssl_threads_cleanup
> > has code like this:
> >
> > if (!GLOBAL_openssl_mutex) return;
> > .....
> > Safefree(GLOBAL_openssl_mutex);
> >
> > and nothing ion the code ever clears GLOBAL_openssl_mutex.
> >
> > so if it is called multiple times, Safefree(GLOBAL_openssl_mutex)
> > will be called multiple times, and that is the cause of the crash
> > reported below.
>
> Well, to play safe we can add after each Safefree a line like this:
>
> Safefree(any_pointer);
> any_pointer = NULL;
>
> It should go to 2 places where we call Safefree(...):
> - openssl_dynlocking_destroy_function()
> - openssl_threads_cleanup()
> (IMHO you can add&commit this as it make things only better)
>
> But in my opinion it is not the core of the problem.
I agree. I think if there can be (and there are) multiple calls to END within
a process, then we need to make sure that END is not really the end of
Net::SSLeay. If it can be called multiple times within a single process, even
during startup, then it should not call
MUTEX_DESTROY(&LIB_init_mutex);
and it should not call openssl_threads_cleanup();
Therefore, I dont think we should have END at all.
Then, you might ask, how will openssl_threads_cleanup() get called? I cant see
a way to call it "just before the process (as opposed to a thread) really
exits". But what would be the problem if we did not call it? A little leak
that is only evident at exit time? There would not be a per-thread leak, and
there are no per-thread resources that we need to free.
Show quoted text>
> > So, perhaps the END function is not required at all?
>
> I do not thik so, maybe the END() is not the best place but we should call
> openssl_threads_cleanup somewhere.
>
> > And openssl_threads_cleanup should protect itself against multiple
> > calls?
>
> Yes this is correct.
>
> > And openssl_threads_cleanup should only be called when the whole
> > interpreter exits (how can we catch that?)
>
> Yes this is correct, normally using END() should assure that we call it just
> once just before perl interpreter is being exited.
I used to think so too, but based on the above I now know and have observed
that not to be the case.
Show quoted text>
> However there are some gotchas connected to mod_perl mode of operation.
>
> Here is my theory (I have not Linux+Apache+mod_perl+mod_ssl+net-ssleay
> enviroment to test it right now):
>
> 1/ The biggest problem is that we have double openssl initialization issue
> - one from Apache's mod_ssl - see httpd-2.2.22\modules\ssl\ssl_util.c -
> functions ssl_util_thread_setup/ssl_util_thread_cleanup
> - the second from Net::SSleay
>
> I am not sure what initialization happes first but I guess that mod_ssl is
> loaded before mod_perl+net-ssleay.
>
> Double initialization is an issue with:
> - multitreading related stuff
> - multiple "SSLeay_add_ssl_algorithms()" calls (you perhaps remember that
> one)
but only when called simultaneously by 2 threads, I think.
Show quoted text>
> I think this is the reason why it crashes - simmply:
> - mod_ssl initializes openssl library (with all thread related mutexes and
> callbacks)
> - Net::SSLeay does openssl initialization in its way again
> - this should inevitably crash
Disagree, the immediate cause of the crash is double free of
GLOBAL_openssl_mutex in openssl_threads_init
I have proved that is what happens, and that if I comment out the
Safefree(GLOBAL_openssl_mutex);
it does not crash.
Show quoted text>
> 2/ I am not an expert on mod_perl but according to
>
http://perl.apache.org/docs/2.0/user/intro/overview.html#Threads_Support it
> is hard to guess how Apache with threaded MPMs works with Net::SSLeay.
>
> According to link above it seems that each Apache thread creates it own perl
> interpret within the same Apache process. I am not sure what exactly
> happens when you load Net::SSleay into it.
>
> Current Net::SSLeay implementation is ready for scenario like this:
> a/ perl starts (as a new process)
> b/ via "use Net::SSLeay" (with one perl interpret is running) we invoke
> BOOT: section of SSLeay.sx
> c/ Boot: section calls openssl_threads_init()
> d/ shortly after "use Net::SSLeay" we usually call
> "SSLeay_add_ssl_algorithms()"
> e/ later in the perl code we create some new threads via
> "threads->new(\&do_the_job)"
>
> Whereas mod_perl scenario is IMHO something like this (not sure about the
> order)
> a/ Apache starts (as a new process)
> b/ Apache loads mod_ssl module (which does all necessary openssl
> initialization)
> c/ Apache creates a couple of MPM threads + starts couple of perl interprets
> d/ Then you try to load Net::SSleay
> e/ Net::SSleay is not aware about the fact that openssl library was already
> initialized and call initialization again
> f/ Note that at this point we have multiple perl interprets - I am not sure
> how many times Net::SSLeay's BOOT: section is gonna be called
> g/ An you are right as there are multiple perl interprets the END(...)
> section is called for each perl interpret (but it should happen IMHO at
> Apache shutdown)
>
> My conclusion:
>
> 1/ If we can somehow detect that thread related setup of openssl lib was
> already done we can simply skip our initialization
I dont believe that is possible, and in any case would not solve the crash we
are seeing now.
Show quoted text>
> 2/ As mod_ssl calls itself SSL_library_init() which is an alias for
> "SSLeay_add_ssl_algorithms()" I am not sure what happens when you call it
> again from perl via Net::SSLeay::SSL_library_init() - I expect a crash
I think we should remove END and openssl_threads_cleanup()
Show quoted text>
>
> --
> kmx
--
Mike McCauley mikem@open.com.au
Open System Consultants Pty. Ltd
9 Bulbul Place Currumbin Waters QLD 4223 Australia
http://www.open.com.au
Phone +61 7 5598-7474 Fax +61 7 5598-7070
Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, NetWare etc.
