Skip Menu |

This queue is for tickets about the SVN-Web CPAN distribution.

Report information
The Basics
Id: 24507
Status: resolved
Priority: 0/
Queue: SVN-Web

People
Owner: Nobody in particular
Requestors: dland [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Critical
Broken in: 0.51
Fixed in: (no value)



Subject: Unbounded memory consumption
I'm running SVN::Web as a persistent process for various mongueurs.net repositories. There are 36 in all. Many are small, only a couple exceed a thousand commits. http://svnweb.mongueurs.net/ Memory consumption increase at around 30Mb per hour until the process dumps core when it hits the 512Mb process limit. I implemented disk caching, but that has not made any difference. My theory is that every page rendered lies around in memory somewhere and is not freed. But I don't know how to test that. Where/how can I instrument the code to fond out where it's leaking? Thanks, David Landgren
Subject: Re: [rt.cpan.org #24507] Unbounded memory consumption
Date: Mon, 22 Jan 2007 22:47:36 +0000
To: bug-SVN-Web [...] rt.cpan.org
From: Nik Clayton <nik [...] ngo.org.uk>
David Landgren via RT wrote: Show quoted text
> I'm running SVN::Web as a persistent process for various mongueurs.net > repositories. There are 36 in all. Many are small, only a couple exceed > a thousand commits. > > http://svnweb.mongueurs.net/ > > Memory consumption increase at around 30Mb per hour until the process > dumps core when it hits the 512Mb process limit. I implemented disk > caching, but that has not made any difference. My theory is that every > page rendered lies around in memory somewhere and is not freed. But I > don't know how to test that.
Damn. I discovered memory leaks in the transition from 0.49 to 0.50, and I thought I'd caught all of them. Here's an example. http://svn.haxx.se/dev/archive-2006-11/0199.shtml I've just committed my test script, at http://jc.ngo.org.uk:80/svnweb/jc/revision/?rev=1300 Grab that, it might help. I was certain that I'd caught all the leaks for 0.50, as the tests (in conjunction with top(1)) definitely showed no leaks. Of course, now I come to re-run them again the leak is definitely there. As I say -- damn. I'll take a look at this tomorrow and see if I can work out where the bug is. N
Dear Nik, thanks for your prompt reply. I must be doing something stupid, because I couldn't get your test script to work: it fails the initial walk of the tree with a template error: svn@profane:~% bin/svnweb-leaker svnadmin: Premature end of content data in dumpstream Created /tmp/uwHaPi2916 SVN::Web now installed! please see config.yaml # First walk Template::process() error: file error - list: not found at /usr/local/lib/perl5/site_perl/5.8.8/SVN/Web.pm line 356. # Looks like your test died before it could output anything. I assume I have to add a key/value pair to the $test->set_config() call, but I'm not sure what. Ideally it should use the same config more or less as the current installation, with all the actions, trac style sheets, log messages filters and so forth. (I didn't state it clearly apart from the "Broken in" field, but this is indeed 0.51 we are talking about here). Thanks for any pointers you can provide, David
Subject: Re: [rt.cpan.org #24507] Unbounded memory consumption
Date: Tue, 23 Jan 2007 10:16:28 +0000
To: bug-SVN-Web [...] rt.cpan.org
From: Nik Clayton <nik [...] ngo.org.uk>
David Landgren via RT wrote: Show quoted text
> (I didn't state it clearly apart from the "Broken in" field, but this is > indeed 0.51 we are talking about here).
S'OK. I'm on the case. I've gone back to branches/svn-client r1249 (where I fixed that I thought was the last leak), and verified that that code does not leak. Or, if it does, it leaks so slowly that it doesn't show up after 10 full site traversals. Since HEAD is at r1300 I should be able to do quick binary search to track down which revision brought the leak back. N
Subject: Re: [rt.cpan.org #24507] Unbounded memory consumption
Date: Tue, 23 Jan 2007 11:53:39 +0100
To: bug-SVN-Web [...] rt.cpan.org
From: David Landgren <david [...] landgren.net>
nik@ngo.org.uk via RT wrote: Show quoted text
> S'OK. I'm on the case. I've gone back to branches/svn-client r1249 (where > I fixed that I thought was the last leak), and verified that that code does > not leak. > > Or, if it does, it leaks so slowly that it doesn't show up after 10 full > site traversals. > > Since HEAD is at r1300 I should be able to do quick binary search to track > down which revision brought the leak back.
Ok, thanks for looking into this. Let me know when you want a guinea pig to try it out in the real world. Thanks, David
Subject: Re: [rt.cpan.org #24507] Unbounded memory consumption
Date: Tue, 23 Jan 2007 11:11:19 +0000
To: bug-SVN-Web [...] rt.cpan.org
From: Nik Clayton <nik [...] ngo.org.uk>
David Landgren via RT wrote: Show quoted text
> thanks for your prompt reply. I must be doing something stupid, because > I couldn't get your test script to work: it fails the initial walk of > the tree with a template error:
OK. Got some alternatives for you to try. First, try the one at http://jc.ngo.org.uk/svnweb/jc/view/nik/CPAN/SVN-Web/branches/svn-client/author.t/memory-leak.t instead. It forks a copy of svnweb-server to test against, which guards against any memory problems in the test script causing issues. You need to run Build.PL before hand, and said that you want to run the web tests. perl Build.PL Build prove -b author.t/memory-leak.t Do that in one window, run top(1) in another window. On my server here the memory use hits about 23,900K and then stays steady. If you see the same that indicates that it's probably not the code exercised by svnweb-server. I've attached two variations on the above test. Instead of using svnweb-server they exercise mod_perl (1) and CGI respectively. If I run the mod_perl1 test locally I see the httpd process continually increase in size. If I run the CGI test locally I don't see any increase (which makes sense, the CGI processes all exit at the end, so no opportunity for leaks). Could you run these as well, and verify that you see the same behaviour. To summarise, I'm expecting: memory-leak.t - no leak modperl-memory-leak.t - show a leak cgi-memory-leak.t - no leak N
#!/usr/bin/perl -w use strict; use Test::More qw(no_plan); use POSIX (); use SVN::Web::Test; use SVN::Web::ConfigData; use LWP::Simple; my $repos = 't/repos'; my $httpd = SVN::Web::ConfigData->config('apache_path'); my $port = SVN::Web::ConfigData->config('httpd_port'); my $libexec_dir = SVN::Web::ConfigData->config('libexec_dir'); my $httpd_version = SVN::Web::ConfigData->config('httpd_version'); my $test = SVN::Web::Test->new(repo_path => 't/repos', repo_dump => 't/test_repo.dump', apache1_path => $httpd, httpd_port => $port); # Create the httpd config file my $template = Template->new(); my $cwd = POSIX::getcwd(); my $dir = $test->install_dir(); $template->process("conf/apache$httpd_version.tt", { blib_dir => "$cwd/blib/lib", svnweb_install_dir => $dir, httpd_port => $port, cgi_bin => 1, libexec_dir => $libexec_dir, }, "$dir/httpd.conf"); undef $template; $test->start_server(qq{ $httpd -f $dir/httpd.conf -X }); $test->mech()->get_ok($test->site_root()); my $url = $test->site_root(); my $mech = $test->mech(); # First, make sure that caching is turned off, and walk the whole tree. # Build a hash that maps URLs to contents. We'll check this later. for (1 .. 100) { diag "$_"; $mech->get($url); $test->walk_site(sub { return; }); } $test->stop_server();
#!/usr/bin/perl -w use strict; use Test::More qw(no_plan); use POSIX (); use SVN::Web::Test; use SVN::Web::ConfigData; use LWP::Simple; my $repos = 't/repos'; my $httpd = SVN::Web::ConfigData->config('apache_path'); my $port = SVN::Web::ConfigData->config('httpd_port'); my $libexec_dir = SVN::Web::ConfigData->config('libexec_dir'); my $mod_perl_path = SVN::Web::ConfigData->config('mod_perl_path'); my $httpd_version = SVN::Web::ConfigData->config('httpd_version'); my $test = SVN::Web::Test->new(repo_path => 't/repos', repo_dump => 't/test_repo.dump', apache1_path => $httpd, httpd_port => $port); # Create the httpd config file my $template = Template->new(); my $cwd = POSIX::getcwd(); my $dir = $test->install_dir(); $template->process("conf/apache$httpd_version.tt", { blib_dir => "$cwd/blib/lib", svnweb_install_dir => $dir, httpd_port => $port, mod_perl => 1, mod_perl_path => $mod_perl_path, libexec_dir => $libexec_dir, }, "$dir/httpd.conf"); undef $template; $test->start_server(qq{ $httpd -f $dir/httpd.conf -X }); $test->mech()->get_ok($test->site_root()); my $url = $test->site_root(); my $mech = $test->mech(); # First, make sure that caching is turned off, and walk the whole tree. # Build a hash that maps URLs to contents. We'll check this later. for (1 .. 100) { diag "$_"; $mech->get($url); $test->walk_site(sub { return; }); } $test->stop_server();
Subject: Re: [rt.cpan.org #24507] Unbounded memory consumption
Date: Tue, 23 Jan 2007 13:54:02 +0100
To: bug-SVN-Web [...] rt.cpan.org
From: David Landgren <david [...] landgren.net>
nik@ngo.org.uk via RT wrote: Show quoted text
> OK. Got some alternatives for you to try. First, try the one at > > http://jc.ngo.org.uk/svnweb/jc/view/nik/CPAN/SVN-Web/branches/svn-client/author.t/memory-leak.t
[...] Show quoted text
> Do that in one window, run top(1) in another window. On my server here the > memory use hits about 23,900K and then stays steady.
I see the same thing here. It hits 29M after initialisation and then does not budge. Show quoted text
> > If you see the same that indicates that it's probably not the code exercised > by svnweb-server. > > I've attached two variations on the above test. Instead of using > svnweb-server they exercise mod_perl (1) and CGI respectively. > > If I run the mod_perl1 test locally I see the httpd process continually > increase in size. If I run the CGI test locally I don't see any increase > (which makes sense, the CGI processes all exit at the end, so no opportunity > for leaks). > > Could you run these as well, and verify that you see the same behaviour.
Having a bit of difficulty trying to get these to run, since my apache+mod_perl binary is compiled statically. I'm currently fiddling with apache1.tt to the damned thing to start up. For what it's worth, my server looks like this: #!/usr/bin/perl use strict; use warnings; use base qw(HTTP::Server::Simple::CGI); use SVN::Web; my $root = '/home/svn/svnweb'; my $port = 8013; sub handle_request { $ENV{SCRIPT_NAME} = 'http://svnweb.mongueurs.net'; print "HTTP/1.1 200 OK\n"; SVN::Web::run_cgi(cgi_class => 'CGI'); } use CGI::Carp qw(fatalsToBrowser); if(!-f "$root/config.yaml") { print <<EOM; Can't find $root/config.yaml. Make sure you've run svnweb-install in $root before running this server. EOM exit; } chdir($root); my $server = __PACKAGE__->new($port); $server->run(); exit; __END__ The port 80 apache just does a simple reverse proxying of this server out to the real world. Hmmm, could it be that SVN::Web is not at fault, but rather HTTP::Server::Simple? I must confess I hadn't thought of that until now. Later, David
Subject: Re: [rt.cpan.org #24507] Unbounded memory consumption
Date: Tue, 23 Jan 2007 13:07:29 +0000
To: bug-SVN-Web [...] rt.cpan.org
From: Nik Clayton <nik [...] ngo.org.uk>
David, david@landgren.net via RT wrote: Show quoted text
>> OK. Got some alternatives for you to try. First, try the one at >> >> http://jc.ngo.org.uk/svnweb/jc/view/nik/CPAN/SVN-Web/branches/svn-client/author.t/memory-leak.t
> > [...] >
>> Do that in one window, run top(1) in another window. On my server here the >> memory use hits about 23,900K and then stays steady.
> > I see the same thing here. It hits 29M after initialisation and then > does not budge.
OK. Good. Show quoted text
>> If you see the same that indicates that it's probably not the code exercised >> by svnweb-server. >> >> I've attached two variations on the above test. Instead of using >> svnweb-server they exercise mod_perl (1) and CGI respectively. >> >> If I run the mod_perl1 test locally I see the httpd process continually >> increase in size. If I run the CGI test locally I don't see any increase >> (which makes sense, the CGI processes all exit at the end, so no opportunity >> for leaks). >> >> Could you run these as well, and verify that you see the same behaviour.
> > Having a bit of difficulty trying to get these to run, since my > apache+mod_perl binary is compiled statically. I'm currently fiddling > with apache1.tt to the damned thing to start up.
OK. If you're running SVN::Web through HTTP::Server::Simple then it's probably not a mod_perl issue, so you can probably ignore that for the time being. BTW, do you use IM? I'm 'nikclayton' on AOL and msn@crf-consulting.co.uk on MSN if you want to chat that way. Show quoted text
> For what it's worth, my server looks like this: > > #!/usr/bin/perl > use strict; > use warnings; > > use base qw(HTTP::Server::Simple::CGI); > use SVN::Web; > > my $root = '/home/svn/svnweb'; > my $port = 8013; > > sub handle_request { > $ENV{SCRIPT_NAME} = 'http://svnweb.mongueurs.net'; > print "HTTP/1.1 200 OK\n"; > SVN::Web::run_cgi(cgi_class => 'CGI'); > } > > use CGI::Carp qw(fatalsToBrowser); > > if(!-f "$root/config.yaml") { > print <<EOM; > Can't find $root/config.yaml. Make sure you've run svnweb-install > in $root before running this server. > EOM > exit; > } > > chdir($root); > my $server = __PACKAGE__->new($port); > $server->run(); > exit; > __END__
OK. That looks close to svnweb-server that ships with SVN::Web. Out of interest, why aren't you using the svnweb-server that ships with SVN::Web? N
On Tue Jan 23 06:31:22 2007, nik@ngo.org.uk wrote: Show quoted text
> David Landgren via RT wrote:
[...] Show quoted text
> OK. Got some alternatives for you to try. First, try the one at
[...] Show quoted text
> If I run the mod_perl1 test locally I see the httpd process > continually > increase in size. If I run the CGI test locally I don't see any > increase > (which makes sense, the CGI processes all exit at the end, so no > opportunity > for leaks). > > Could you run these as well, and verify that you see the same > behaviour.
I succeeded in adapting these latter two scripts to my setup, and I confirm I see leaks in the mod_perl version, between 15 to 60 KB per second. The cgi version does not leak. Show quoted text
> To summarise, I'm expecting: > > memory-leak.t - no leak > modperl-memory-leak.t - show a leak > cgi-memory-leak.t - no leak
I confirm the same behaviour here. Regards, David
On Tue Jan 23 08:08:38 2007, nik@ngo.org.uk wrote: Show quoted text
> OK. That looks close to svnweb-server that ships with SVN::Web. > > Out of interest, why aren't you using the svnweb-server that ships > with > SVN::Web?
It probably was, originally, but has been tweaked since. As for IM, I can do IRC but that's about it. David
The following patch allows you to specify the hostname. This is handy when you want to reverse proxy the output through Apache, and don't want localhost:$port leaking out into the real world. Needs documentation :) David --- bin/svnweb-server Mon Jan 15 18:49:20 2007 +++ /usr/local/bin/svnweb-server Tue Jan 23 14:51:09 2007 @@ -1,4 +1,7 @@ -#!/usr/bin/perl +#!/usr/local/bin/perl + +eval 'exec /usr/local/bin/perl -S $0 ${1+"$@"}' + if 0; # not running under some shell # -*- Mode: cperl; cperl-indent-level: 4 -*- @@ -63,6 +66,7 @@ 'root=s' => \$options{root}, 'port=i' => \$options{port}, 'net-server=s' => \$options{net_server}, + 'hostname=s' => \$options{hostname}, ); sub handle_request { @@ -76,7 +80,12 @@ print <FILE>; close FILE; } else { + if ($options{hostname}) { + $ENV{SCRIPT_NAME} = "http://$options{hostname}"; + } + else { $ENV{SCRIPT_NAME} = 'http://' . $self->host() . ':' . $self->port(); + } print "HTTP/1.1 200 OK\n"; SVN::Web::run_cgi(cgi_class => 'CGI'); }
I have replaced all the mod_perl/fastcgi/standalone code with Plack. Please create a new bug if this bug persists in this environment.