Skip Menu |

This queue is for tickets about the Sphinx-Search CPAN distribution.

Report information
The Basics
Id: 70760
Status: resolved
Priority: 0/
Queue: Sphinx-Search

People
Owner: Nobody in particular
Requestors: kholodkov [...] mail.ru
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: persistent connection problem
Date: Tue, 06 Sep 2011 16:28:52 +0400
To: bug-Sphinx-Search [...] rt.cpan.org
From: Алексей Холодков <kholodkov [...] mail.ru>
Hi, Thanks for your modules. I try to use persistent connection  in Sphinx::Search module $VERSION = '0.240.1' .   Linux. perl v5.8.8 I think that other versions have the same problem. It makes connecction to sphinx and keep it.  But, when I run indexer with option --rotate and reindex data on sphinx server I catch a littele problem. Some first queries fails with: "read failed:" and than it works fine (but not in persistent connection mode). sample code with comments: .... my $sph = Sph::Search->new(); $sph->SetServer($host, $port); $sph->SetConnectTimeout($timeout); $sph->Open(); .... my $res = $sph->Query('test', 'test');  #this is ok  </span>$res = $sph->Query('test', 'test'); #this is ok  and work with the same socket while(<STDIN>){  # wait for reindex data on server     last if $_ =~/\n/; } </span>$res = $sph->Query('test', 'test'); #this fails $res = $sph->Query('test', 'test'); #this is ok, but from this point no persistent connection .....all other queries get new sockets. end code... Is it possible to fix persistent connection and restore it  on fail? and if query falls in persistent mode try to reconnect and resent failed query. Thanks. WBR. Alexey.
Subject: Re: [rt.cpan.org #70760] AutoReply: persistent connection problem
Date: Thu, 15 Sep 2011 13:59:00 +0400
To: bug-Sphinx-Search [...] rt.cpan.org
From: Алексей Холодков <kholodkov [...] mail.ru>
I make some changes in module $VERSION = '0.240.1' to fix the problem: ============================= 1. add two items to object _persistent => undef, 'flag - is persistent connection mode' _connectretries => 1; 'number of reconnections if connection fails' 2. add new method ' SetConnectRetries ' it changes value of _connectretries; 3. _ProcessRequest - send/resend requests to sphinx. 4. Persistent connect data moved to method _Connect ================================================== DIFF BELOW ================================================ 281a282,283 Show quoted text
> _persistent => undef, > _connectretries => 1,
490a493,510 Show quoted text
> =head2 SetConnectRetries > > $sph->SetConnectRetries($retries) > > Set server connection retries (in case of connection fail). > > Returns $sph. > > =cut > > sub SetConnectRetries { > my $self = shift; > my $retires = shift; > croak("connectretries is not numeric") unless ($retires =~ m/$num_re/); > $self->{connectretries} = $retires; > } > >
497,501c517,524 < $fp->write($data); return 1; < if ($fp->eof || ! $fp->write($data)) { < $self->_Error("connection unexpectedly closed (timed out?): $!"); < $self->{_connerror} = 1; < return 0; --- Show quoted text
> unless ( send($fp,$data,0)){ > $self->_Error("connection unexpectedly closed (timed out?): $!"); > $self->{_connerror} = 1; > if ($self->{_socket}) { > close($self->{_socket}); > undef $self->{_socket}; > } > return 0;
510c533 < --- Show quoted text
> $self->_Error(); #reset old errors in new connection
566,567c589,593 < $self->_Send($fp, pack("N", 1)) or return 0; < --- Show quoted text
> unless ($self->_Send($fp, pack("N", 1))) { > $self->{_connerror} = 1; > $self->_Error("error while send version"); > return 0; > }
569c595,604 < --- Show quoted text
> > if ($self->{_persistent}) { > my $req = pack("nnNN", SEARCHD_COMMAND_PERSIST, 0, 4, 1); > unless ($self->_Send($fp, $req)) { > $self->{_connerror} = 1; > $self->_Error("error while set persistent connection"); > return 0; > } > $self->{_socket} = $fp; > }
580d614 < 582,585c616,622 < defined($fp->read($header, 8, 0)) or do { < $self->_Error("read failed: $!"); < return 0; < }; --- Show quoted text
> my $resp = $fp->read($header, 8, 0); > if (!defined($resp) || $resp==0) { > close $self->{_socket}; > undef $self->{_socket}; > $self->_Error("read failed: $!"); > return 0; > }
593c630 < my $response = q{}; --- Show quoted text
> my $response = q{};
638a676,696 Show quoted text
> #----------------------------------------------- > # connect to searchd, send request and get data > > sub _ProcessRequest { > my ($self,$req) = @_; > return unless $req; > my $tries = $self->{_connectretries} + 1; > while( $tries-- ) { > my $fp = $self->_Connect; > if (! $fp) { > next if $self->IsConnectError; > last; > } > $self->_Send($fp, $req) or next; > my $response = $self->_GetResponse ($fp, VER_COMMAND_SEARCH); > return $response if $response; > } > $self->_Error($self->GetLastError . "... ConnectRetries exceed...") if $self->IsConnectError; > return 0; > } >
1525,1526d1582 < my $fp = $self->_Connect() or do { $self->{_reqs} = []; return }; < 1533,1534c1589 < $self->_Send($fp, $req); < --- Show quoted text
> my $response = $self->_ProcessRequest($req);
1536,1537d1590 < < my $response = $self->_GetResponse ( $fp, VER_COMMAND_SEARCH ); 1736d1788 < my $fp = $self->_Connect() or return; 1801,1803c1853,1854 < $self->_Send($fp, $req); < < my $response = $self->_GetResponse($fp, VER_COMMAND_EXCERPT) or return; --- Show quoted text
> my $response = $self->_ProcessRequest($req); > return unless $response;
1855d1905 < my $fp = $self->_Connect() or return; 1867,1868c1917 < $self->_Send($fp, $req); < my $response = $self->_GetResponse ( $fp, VER_COMMAND_KEYWORDS ); --- Show quoted text
> my $response = $self->_ProcessRequest($req);
2008d2056 < my $fp = $self->_Connect() or return; 2011c2059 < send ( $fp, $req, 0); --- Show quoted text
> my $response = $self->_ProcessRequest($req);
2013d2060 < my $response = $self->_GetResponse ( $fp, VER_COMMAND_UPDATE ); 2037c2084 < --- Show quoted text
> $self->{_persistent} = 1;
2043,2047d2089 < < my $req = pack("nnNN", SEARCHD_COMMAND_PERSIST, 0, 4, 1); < $self->_Send($fp, $req) or return 0; < < $self->{_socket} = $fp; 2062a2105 Show quoted text
> $self->{_persistent} = 0;
2088,2089d2130 < < my $fp = $self->_Connect() or return; 2092,2093c2133,2134 < $self->_Send($fp, $req) or return; < my $response = $self->_GetResponse ( $fp, VER_COMMAND_STATUS ); --- Show quoted text
> > my $response = $self->_ProcessRequest($req);
2124,2125d2164 < < my $fp = $self->_Connect() or return; 2128,2129c2167 < $self->_Send($fp, $req) or return; < my $response = $self->_GetResponse ( $fp, VER_COMMAND_FLUSHATTRS ); --- Show quoted text
> my $response = $self->_ProcessRequest($req);
========================================== END DIFF ======================================== Will you insert this implement to module? WBR. Aleksey. 06 сентября 2011, 16:29 от "Bugs in Sphinx-Search via RT" <bug-Sphinx-Search@rt.cpan.org>: Show quoted text
> > Greetings, > > This message has been automatically generated in response to the > creation of a trouble ticket regarding: > "persistent connection problem", > a summary of which appears below. > > There is no need to reply to this message right now. Your ticket has been > assigned an ID of [rt.cpan.org #70760]. Your ticket is accessible > on the web at: > > https://rt.cpan.org/Ticket/Display.html?id=70760 > > Please include the string: > > [rt.cpan.org #70760] > > in the subject line of all future correspondence about this issue. To do so, > you may reply to this message. > > Thank you, > bug-Sphinx-Search@rt.cpan.org > > ------------------------------------------------------------------------- > Hi, > Thanks for your modules. > I try to use persistent connection  in Sphinx::Search module $VERSION = '0.240.1' .   Linux. perl v5.8.8 > I think that other versions have the same problem. > It makes connecction to sphinx and keep it.  > But, when I run indexer with option --rotate and reindex data on sphinx server I catch a littele problem. > Some first queries fails with: "read failed:" and than it works fine (but not in persistent connection mode). > sample code with comments: > .... > my $sph = Sph::Search->new(); > $sph->SetServer($host, $port); > $sph->SetConnectTimeout($timeout); > $sph->Open(); > .... > my $res = $sph->Query('test', 'test');  #this is ok  > </span>$res = $sph->Query('test', 'test'); #this is ok  and work with the same socket > > while(<STDIN>){  # wait for reindex data on server >     last if $_ =~/\n/; > } > </span>$res = $sph->Query('test', 'test'); #this fails > $res = $sph->Query('test', 'test'); #this is ok, but from this point no persistent connection > .....all other queries get new sockets. > > end code... > > Is it possible to fix persistent connection and restore it  on fail? > and if query falls in persistent mode try to reconnect and resent failed query. > > Thanks. > WBR. Alexey. > >
Hi Alexey, Thankyou for providing a patch. I cannot apply it cleanly by pasting from RT. Could you please create the patch with 'diff -u' and attach it as a text file? Are you able to write a test case that covers the problem you were having? Regards, -- Jon Schutz On Thu Sep 15 05:59:17 2011, kholodkov@mail.ru wrote: Show quoted text
> I make some changes in module $VERSION = '0.240.1' to fix the > problem: > > ============================= > 1. add two items to object > _persistent => undef, 'flag - is persistent > connection mode' > _connectretries => 1; 'number of reconnections if > connection fails' > 2. add new method ' SetConnectRetries ' it changes value of > _connectretries; > > 3. _ProcessRequest - send/resend requests to sphinx. > > 4. Persistent connect data moved to method _Connect > ================================================== > DIFF BELOW > ================================================ > 281a282,283
> > _persistent => undef, > > _connectretries => 1,
> 490a493,510
> > =head2 SetConnectRetries > > > > $sph->SetConnectRetries($retries) > > > > Set server connection retries (in case of connection fail). > > > > Returns $sph. > > > > =cut > > > > sub SetConnectRetries { > > my $self = shift; > > my $retires = shift; > > croak("connectretries is not numeric") unless ($retires =~
> m/$num_re/);
> > $self->{connectretries} = $retires; > > } > > > >
> 497,501c517,524 > < $fp->write($data); return 1; > < if ($fp->eof || ! $fp->write($data)) { > < $self->_Error("connection unexpectedly closed (timed out?): > $!"); > < $self->{_connerror} = 1; > < return 0; > ---
> > unless ( send($fp,$data,0)){ > > $self->_Error("connection unexpectedly closed (timed out?):
> $!");
> > $self->{_connerror} = 1; > > if ($self->{_socket}) { > > close($self->{_socket}); > > undef $self->{_socket}; > > } > > return 0;
> 510c533 > < > ---
> > $self->_Error(); #reset old errors in new connection
> 566,567c589,593 > < $self->_Send($fp, pack("N", 1)) or return 0; > < > ---
> > unless ($self->_Send($fp, pack("N", 1))) { > > $self->{_connerror} = 1; > > $self->_Error("error while send version"); > > return 0; > > }
> 569c595,604 > < > ---
> > > > if ($self->{_persistent}) { > > my $req = pack("nnNN", SEARCHD_COMMAND_PERSIST, 0, 4, 1); > > unless ($self->_Send($fp, $req)) { > > $self->{_connerror} = 1; > > $self->_Error("error while set persistent connection"); > > return 0; > > } > > $self->{_socket} = $fp; > > }
> 580d614 > < > 582,585c616,622 > < defined($fp->read($header, 8, 0)) or do { > < $self->_Error("read failed: $!"); > < return 0; > < }; > ---
> > my $resp = $fp->read($header, 8, 0); > > if (!defined($resp) || $resp==0) { > > close $self->{_socket}; > > undef $self->{_socket}; > > $self->_Error("read failed: $!"); > > return 0; > > }
> 593c630 > < my $response = q{}; > ---
> > my $response = q{};
> 638a676,696
> > #----------------------------------------------- > > # connect to searchd, send request and get data > > > > sub _ProcessRequest { > > my ($self,$req) = @_; > > return unless $req; > > my $tries = $self->{_connectretries} + 1; > > while( $tries-- ) { > > my $fp = $self->_Connect; > > if (! $fp) { > > next if $self->IsConnectError; > > last; > > } > > $self->_Send($fp, $req) or next; > > my $response = $self->_GetResponse ($fp,
> VER_COMMAND_SEARCH);
> > return $response if $response; > > } > > $self->_Error($self->GetLastError . "... ConnectRetries
> exceed...") if $self->IsConnectError;
> > return 0; > > } > >
> 1525,1526d1582 > < my $fp = $self->_Connect() or do { $self->{_reqs} = []; return > }; > < > 1533,1534c1589 > < $self->_Send($fp, $req); > < > ---
> > my $response = $self->_ProcessRequest($req);
> 1536,1537d1590 > < > < my $response = $self->_GetResponse ( $fp, VER_COMMAND_SEARCH ); > 1736d1788 > < my $fp = $self->_Connect() or return; > 1801,1803c1853,1854 > < $self->_Send($fp, $req); > < > < my $response = $self->_GetResponse($fp, VER_COMMAND_EXCERPT) > or return; > ---
> > my $response = $self->_ProcessRequest($req); > > return unless $response;
> 1855d1905 > < my $fp = $self->_Connect() or return; > 1867,1868c1917 > < $self->_Send($fp, $req); > < my $response = $self->_GetResponse ( $fp, VER_COMMAND_KEYWORDS > ); > ---
> > my $response = $self->_ProcessRequest($req);
> 2008d2056 > < my $fp = $self->_Connect() or return; > 2011c2059 > < send ( $fp, $req, 0); > ---
> > my $response = $self->_ProcessRequest($req);
> 2013d2060 > < my $response = $self->_GetResponse ( $fp, VER_COMMAND_UPDATE ); > 2037c2084 > < > ---
> > $self->{_persistent} = 1;
> 2043,2047d2089 > < > < my $req = pack("nnNN", SEARCHD_COMMAND_PERSIST, 0, 4, 1); > < $self->_Send($fp, $req) or return 0; > < > < $self->{_socket} = $fp; > 2062a2105
> > $self->{_persistent} = 0;
> 2088,2089d2130 > < > < my $fp = $self->_Connect() or return; > 2092,2093c2133,2134 > < $self->_Send($fp, $req) or return; > < my $response = $self->_GetResponse ( $fp, VER_COMMAND_STATUS ); > ---
> > > > my $response = $self->_ProcessRequest($req);
> 2124,2125d2164 > < > < my $fp = $self->_Connect() or return; > 2128,2129c2167 > < $self->_Send($fp, $req) or return; > < my $response = $self->_GetResponse ( $fp, VER_COMMAND_FLUSHATTRS > ); > ---
> > my $response = $self->_ProcessRequest($req);
> > ========================================== > END DIFF > ======================================== > > Will you insert this implement to module? > > WBR. Aleksey. > > > > > > > 06 сентября 2011, 16:29 от "Bugs in Sphinx-Search via RT" <bug-Sphinx- > Search@rt.cpan.org>:
> > > > Greetings, > > > > This message has been automatically generated in response to the > > creation of a trouble ticket regarding: > > "persistent connection problem", > > a summary of which appears below. > > > > There is no need to reply to this message right now. Your ticket
> has been
> > assigned an ID of [rt.cpan.org #70760]. Your ticket is accessible > > on the web at: > > > > https://rt.cpan.org/Ticket/Display.html?id=70760 > > > > Please include the string: > > > > [rt.cpan.org #70760] > > > > in the subject line of all future correspondence about this issue.
> To do so,
> > you may reply to this message. > > > > Thank you, > > bug-Sphinx-Search@rt.cpan.org > > > >
> -------------------------------------------------------------------------
> > Hi, > > Thanks for your modules. > > I try to use persistent connection  in Sphinx::Search module
> $VERSION = '0.240.1' .   Linux. perl v5.8.8
> > I think that other versions have the same problem. > > It makes connecction to sphinx and keep it.  > > But, when I run
> indexer with option --rotate and reindex data on sphinx server I catch > a littele problem.
> > Some first queries fails with: "read failed:" and than it works fine
> (but not in persistent connection mode).
> > sample code with comments: > > .... > > my $sph = Sph::Search->new(); > > $sph->SetServer($host, $port); > > $sph->SetConnectTimeout($timeout); > > $sph->Open(); > > .... > > my $res = $sph->Query('test', 'test');  #this is ok  > > </span>$res =
> $sph->Query('test', 'test'); #this is ok  and work with the same > socket
> > > > while(<STDIN>){  # wait for reindex data on server > >     last if $_ =~/\n/; > > } > > </span>$res = $sph->Query('test', 'test'); #this fails > > $res = $sph->Query('test', 'test'); #this is ok, but from this point
> no persistent connection
> > .....all other queries get new sockets. > > > > end code... > > > > Is it possible to fix persistent connection and restore it  on fail? > > and if query falls in persistent mode try to reconnect and resent
> failed query.
> > > > Thanks. > > WBR. Alexey. > > > >
Subject: Re: [rt.cpan.org #70760] persistent connection problem
Date: Mon, 19 Sep 2011 11:46:56 +0400
To: bug-Sphinx-Search [...] rt.cpan.org
From: holodkov [...] corp.mail.ru
Hi Jon, I can't write perl test which can show this problem because of shinx (or emulator) needed while test. But i write simple script whith comment (in attachments). You need to add sphinx host, port and test query to this script. Script open persistent connection, makes 4 queries to sphinx and than wait. You should broke connection( reindex with rotate or stop and start sphinx) than press "Enter". The script makes 6 more queries. if request successful it prints total found, else Last_Error. You can use 2 ways to see problem use trace and look on system calls to find open ports. Or just add print $fp."\n"; in _GetResponse function in Sphinx::Search module. It helps you to see descriptor of open port. Than you'll see that first 4 queries opens only 1 descriptor,(this is correct), but after stop/start sphinx 6-th query falls with read failed and 7-10 open's their own ports for queries(it's not correct for persistent mode). after patch module open persistent connection after fail and resend last query and 6-10 queries open only 1 port. You can find 2 files in attachments(patch and simple script) ./Search.pm - old module ./Sphinx/Search.pm -new WBR. Alexey. 16 сентября 2011, 04:36 от "Jon Schutz via RT" <bug-Sphinx-Search@rt.cpan.org>: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=70760 > > > Hi Alexey, > > Thankyou for providing a patch. I cannot apply it cleanly by pasting > from RT. Could you please create the patch with 'diff -u' and attach it > as a text file? > > Are you able to write a test case that covers the problem you were having? > > Regards, > > -- > > Jon Schutz >

Message body is not shown because sender requested not to inline it.

Download patch.sphinx_module
application/octet-stream 7.2k

Message body not shown because it is not plain text.

Subject: Re: [rt.cpan.org #70760] persistent connection problem
Date: Mon, 28 Nov 2011 10:40:02 +0400
To: bug-Sphinx-Search [...] rt.cpan.org
From: Алексей Холодков <kholodkov [...] mail.ru>
Hi, Jon, Is it possible to fix this issue? WBR. Aleksey.
Subject: Re: [rt.cpan.org #70760] persistent connection problem
Date: Mon, 28 Nov 2011 17:16:02 +1030
To: bug-Sphinx-Search [...] rt.cpan.org
From: Jon Schutz <jon.schutz [...] youramigo.com>
Sorry, it's been delayed as I've been fully committed on other tasks. Unfortunately it's not as simple as applying your patch as I'd need to rework the patch for the latest version and then test. If you had a tested patch against 0.26.1 I might be able to get it out quicker. Thanks again for doing the patching. On 28/11/11 17:10, Алексей Холодков via RT wrote: Show quoted text
> Queue: Sphinx-Search > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=70760 > > > Hi, Jon, > Is it possible to fix this issue? > WBR. Aleksey.
-- Jon Schutz CTO, YourAmigo Ltd 53 Gilbert St Adelaide SA 5000 Ph: +61 8 82119211 Fax: +61 8 8211 6356 http://www.youramigo.com
Subject: Re[2]: [rt.cpan.org #70760] persistent connection problem
Date: Sat, 10 Mar 2012 18:44:05 +0400
To: bug-Sphinx-Search [...] rt.cpan.org
From: Алексей Холодков <kholodkov [...] mail.ru>
Hi, Jon, Sorry for delay. I attach patch file for module 0.27.1, the latesv version at the moment. will you run test's on resulting module? the changes the same as in previous version. WBR, Aleksey. 28 ноября 2011, 10:46 от "jon.schutz@youramigo.com via RT" <bug-Sphinx-Search@rt.cpan.org>: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=70760 > > > Sorry, it's been delayed as I've been fully committed on other tasks. > Unfortunately it's not as simple as applying your patch as I'd need to > rework the patch for the latest version and then test. If you had a > tested patch against 0.26.1 I might be able to get it out quicker. > > Thanks again for doing the patching. > > On 28/11/11 17:10, Алексей Холодков via RT wrote:
> > Queue: Sphinx-Search > > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=70760 > > > > > Hi, Jon, > > Is it possible to fix this issue? > > WBR. Aleksey.
> > -- > Jon Schutz > CTO, YourAmigo Ltd > 53 Gilbert St > Adelaide SA 5000 > Ph: +61 8 82119211 > Fax: +61 8 8211 6356 > http://www.youramigo.com > >
Download patch.027.1
application/octet-stream 7.4k

Message body not shown because it is not plain text.

Subject: Re: [rt.cpan.org #70760] persistent connection problem
Date: Sun, 18 Mar 2012 18:08:20 +1030
To: bug-Sphinx-Search [...] rt.cpan.org
From: Jon Schutz <jon.schutz [...] youramigo.com>
Hi Alexey, I have incorporated your patch with a minor change to _ProcessRequest - it needs the correct client version for the type of command it's handling. Version 0.27.2 has been uploaded to CPAN. Thanks for your contribution. Regards, -- Jon Schutz CTO, YourAmigo Ltd 53 Gilbert St Adelaide SA 5000 Ph: +61 8 82119211 Fax: +61 8 8211 6356 http://www.youramigo.com On 03/11/2012 01:14 AM, Алексей Холодков via RT wrote: Show quoted text
> Queue: Sphinx-Search > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=70760 > > > Hi, Jon, > > Sorry for delay. > I attach patch file for module 0.27.1, the latesv version at the moment. > will you run test's on resulting module? > the changes the same as in previous version. > > WBR, Aleksey. > > 28 ноября 2011, 10:46 от "jon.schutz@youramigo.com via RT" <bug-Sphinx-Search@rt.cpan.org>:
>> <URL: https://rt.cpan.org/Ticket/Display.html?id=70760 > >> >> Sorry, it's been delayed as I've been fully committed on other tasks. >> Unfortunately it's not as simple as applying your patch as I'd need to >> rework the patch for the latest version and then test. If you had a >> tested patch against 0.26.1 I might be able to get it out quicker. >> >> Thanks again for doing the patching. >> >> On 28/11/11 17:10, Алексей Холодков via RT wrote:
>>> Queue: Sphinx-Search >>> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=70760 > >>> >>> Hi, Jon, >>> Is it possible to fix this issue? >>> WBR. Aleksey.
>> -- >> Jon Schutz >> CTO, YourAmigo Ltd >> 53 Gilbert St >> Adelaide SA 5000 >> Ph: +61 8 82119211 >> Fax: +61 8 8211 6356 >> http://www.youramigo.com >> >>