Skip Menu |

This queue is for tickets about the CGI-Application-Plugin-AnyTemplate CPAN distribution.

Report information
The Basics
Id: 18157
Status: resolved
Priority: 0/
Queue: CGI-Application-Plugin-AnyTemplate

People
Owner: Nobody in particular
Requestors: dan.horne [...] redbone.co.nz
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 0.17
Fixed in: 0.18



Subject: CAP::AnyTemplate uses huge amounts of memory with HTML::Template::Expr
Hello CGI::Application::Plugin::Anytemplate with HTML::Template::Expr uses a heck of a lot of memory - growing to 100s MB when outputting templates in loop. My environment: Parse::RecDescent 1.94 HTML::Template 2.8 HTML::Template::Expr 0.06 CGI::Application::Plugin::AnyTemplate 0.17 Test case: TestExpr.pm =========== package TestExpr; use base 'CGI::Application'; use CGI::Application::Plugin::AnyTemplate; use Data::Dumper; use strict; sub cgiapp_init { my $self = shift; $self->template->config( default_type => 'HTMLTemplateExpr', ); } sub setup { my $self = shift; $self->start_mode('test_case'); $self->run_modes(['test_case']); } sub test_publish { my $self = shift; my $template = $self->template->load(file => 'test'); $template->param('title', 'This is my title'); $template->param('summary', 'Just some summary text'); # not required, but this illustrates what I'm doing ... publishing # to the filsystem open (my $fh, '>', '/tmp/output.html'); print $fh ${$template->output}; close $fh; 1; } sub test_case { my $self = shift; foreach my $j (0..50000) { $self->test_publish(); } return 'ok'; } 1; instance.pl - run from the command line =========== #!/usr/local/bin/perl -w use strict; use CGI::Carp qw(fatalsToBrowser); use CGI::Application::Dispatch; CGI::Application::Dispatch->dispatch( CGIAPP_DISPATCH_DEFAULT => 'test_case', CGIAPP_DISPATCH_RM => 1, TABLE => { test_case => 'TestExpr', } ); test.html ========= <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html lang="en"> <head> <title>HTML::Template::Expr Test</title> </head> <body> Title: <tmpl_var title> Summary: <tmpl_var summary> </body> </html> regards Dan
Subject: Re: [rt.cpan.org #18157] CAP::AnyTemplate uses huge amounts of memory with HTML::Template::Expr
Date: Mon, 13 Mar 2006 22:36:13 -0500
To: bug-CGI-Application-Plugin-AnyTemplate [...] rt.cpan.org
From: Michael Graham <magog [...] the-wire.com>
Hi Dan, Thanks for the CAP::AnyTemplate request. Just to let you know - I'm not ignoring your ticket, but I'm swamped at work for the next few weeks and I won't have the time to investigate this very quickly. Sorry about that! For my purposes, I mostly use TT and a custom driver for a legacy templating system we use where I work. I don't experience any crazy memory usage (that I know of). The TT driver uses a cache so that it doesn't call Template->new all the time. The HT and HT::Expr drivers don't do this. But that's because in TT land, the workflow is: $tt = Template->new; $tt->process($file1, \%params, \$output1); $tt->process($file2, \%params, \$output2); $tt->process($file3, \%params, \$output3); Whereas in HT land, the workflow is: $tmpl = HTML::Template::Expr->new(filename => $file1); $tmpl->param(%params); $output1 = $tmpl->output; $tmpl = HTML::Template::Expr->new(filename => $file2); $tmpl->param(%params); $output1 = $tmpl->output; $tmpl = HTML::Template::Expr->new(filename => $file3); $tmpl->param(%params); $output1 = $tmpl->output; So CAP::AnyTemplate does indeed create a new HTML::Template::Expr object for each filename. Which is good and proper. I don't believe there's any way of avoiding this, since you can't change the filename of an HTML::Template object after the fact. What is probably happening is that AnyTemplate's object is failing to go out of scope (due to a circular reference somewhere), and so unused HTML::Template parser objects are piling up in memory without being garbage collected. It would be interesting to see if you can reproduce the memory leak with the TT driver. Even if the leak is smaller, it would still shed some light. I will look into this when I have some time, but, as I say, it will take me a while. In the meantime, if you are able to make any headway, please let me know Michael --- Michael Graham <magog@the-wire.com>
From: Dan Horne
Hi Michael, thanks for the update. Like yourself, I'm also pretty snowed under with work for the next few weeks, so there isn't much that I am able to do beyond trying to track down the memory leak. I have had a hunt for circular references, but alas, nothing has jumped out at me. I have determined that the memory usage is most prevalent in HTML::Template::Expr as it assigns a Parse::RecDescent object to a global variable ($HTML::Template::Expr::PARSER), and this seems to take up a lot of memory. Neither HTML::Template nor HTML::Template::Pluggable exhibit the extremes of the memory issue, but unfortunately, I can't seem to get them to work with all of my embedded templates (thanks for the embed feature – it is used extensively) I the meantime, I seem to have a workaround – I just undef $template->{driver} in my scripts prior to $template going out of scope. It's an ugly solution, but I'm trying to avoid switching to TT as we have too many templates to change at the moment, and I think the HTML designers would throttle me if I introduced a new template system to them. Anyway, once my deadlines have passed in early April, I'll have some more time to devote to this. Regards Dan
Subject: Re: [rt.cpan.org #18157] CAP::AnyTemplate uses huge amounts of memory with HTML::Template::Expr
Date: Tue, 14 Mar 2006 15:30:19 -0500
To: bug-CGI-Application-Plugin-AnyTemplate [...] rt.cpan.org
From: Michael Graham <magog [...] the-wire.com>
Hi Dan, I'm glad you have found a workaround - that takes some of the pressure off for now. Show quoted text
> exhibit the extremes of the memory issue, but unfortunately, I can't > seem to get them to work with all of my embedded templates
Hmmm... If this leak only manifests itself when the embed feature is extensively used, that might point to the source of the circular reference. In fact, looking closer, I bet that's it. In ComponentHandler.pm: sub new { my $proto = shift; my $class = ref $proto || $proto; my %args = @_; my $self = {}; bless $self, $class; $self->{'webapp'} = $args{'webapp'}; $self->{'containing_template'} = $args{'containing_template'}; weaken $self->{'webapp'}; return $self; } The $self->{'webapp'} reference is weakened but the $self->{'contaning_template'} reference isn't. It probably should be. I'll have to try weakening it and seeing if all the tests still pass. Michael --- Michael Graham <magog@the-wire.com>
Subject: RE: [rt.cpan.org #18157] CAP::AnyTemplate uses huge amounts of memory with HTML::Template::Expr
Date: Wed, 15 Mar 2006 11:11:31 +1300
To: <bug-CGI-Application-Plugin-AnyTemplate [...] rt.cpan.org>
From: "Dan Horne" <dan.horne [...] redbone.co.nz>
Hi Michael Show quoted text
> Hmmm... If this leak only manifests itself when the embed feature is > extensively used, that might point to the source of the circular > reference
Alas, the test I logged with the ticket exhibits this issue, but does not use embed. My problem was that I couldn't switch to the HTML::Template or HTML::Template::Pluggable drivers, as they had problems with my extensive use of the embed functionality. I will try weakening $self->{'containing_template'} however, and see what happens Thanks for you help Dan
From: Dan Horne
Hi Michael FYI, I added weaken $self->{'containing_template'}; to ComponentHandler.pm as per your last note, and both the test case without any embeds and our system with embeds seem to be working as expected, but without consuming huge amounts of memory. There may need to be some more through testing involved, but as far as we can tell, the problem has disappeared with this change. Thanks Dan
Hi Dan, This is a response to your (very old) ticket regarding CAP-AnyTemplate. For this new release (0.18), I made the recommended change to the ComponentHandler module. Hopefully this resolves the memory issues with HTML::Template::Expr. Thank you for your report and your patience! :) Michael